MATCH reports the metrics P@1, P@3, P@5, nDCG@3, and nDCG@5 using the predicted labels with the top k probability scores. We recognize that a paper can have an arbitrary number of biomimicry functions associated with it. Instead of fixing a constant k such that the top k predicted labels for every paper are returned, we allow the user to specify a threshold θ between 0 and 1 such that MATCH will treat every label whose predicted probability exceeds θ as a positive prediction, and every other label as a negative prediction.
However, we also recognize that the goal of this work on the PeTaL automatic labeling system is not the labeler itself, but a recommendation system. Even if the PeTaL labeler classifies millions of biology papers with perfect precision and recall, it will produce tens of thousands of papers for each biomimicry function. In order to best recommend a biology research paper to an engineer, we require some method of ranking the papers by relevance to the biomimicry function. In this case, the distinction between top k-based selection and threshold-based selection is immaterial and we identify the actual “confidence scores” that MATCH produces for each label as the desired output of our labeling system.
bruffridge
commented
3 years ago
MATCH reports the metrics P@1, P@3, P@5, nDCG@3, and nDCG@5 using the predicted labels with the top k probability scores. We recognize that a paper can have an arbitrary number of biomimicry functions associated with it. Instead of fixing a constant k such that the top k predicted labels for every paper are returned, we allow the user to specify a threshold θ between 0 and 1 such that MATCH will treat every label whose predicted probability exceeds θ as a positive prediction, and every other label as a negative prediction.
However, we also recognize that the goal of this work on the PeTaL automatic labeling system is not the labeler itself, but a recommendation system. Even if the PeTaL labeler classifies millions of biology papers with perfect precision and recall, it will produce tens of thousands of papers for each biomimicry function. In order to best recommend a biology research paper to an engineer, we require some method of ranking the papers by relevance to the biomimicry function. In this case, the distinction between top k-based selection and threshold-based selection is immaterial and we identify the actual “confidence scores” that MATCH produces for each label as the desired output of our labeling system.