search

allenai / allennlp

An open-source NLP research library, built on PyTorch.
http://www.allennlp.org
Apache License 2.0
11.72k stars 2.25k forks source link

Metrics for Multi-label Classification Task #4557

Open shounakpaul95 opened 3 years ago

shounakpaul95 commented 3 years ago

Is there any built-in functionality to calculate metrics like Precision, Recall and F1-Score for the multi-label multi-class classification scenario (one or more target classes)?

The current implementation of FBeta Measure can only take inputs for the single-label scenario.

JohnGiorgi commented 3 years ago

We (not affiliated with AllenNLP) are actually working on exactly that! Our implementation subclasses FBetaMeasure and has (almost) all the same unit tests. Still stuck on a couple.

You can take a look at it here where we also have a multi-label dataset reader and a multi-label text classification model. The metric specifically is here.

If the AllenNLP devs think this is something that should be added to allennlp or allennlp-models I would be happy to work on a PR.

tanmayag78 commented 3 years ago

I have made my custom function for calculating Precision, Recall, and F1-Score for the multi-label classification. It's a generic function that can be used with any algo or framework you are using for multilabel classification.

dirkgr commented 3 years ago

@JohnGiorgi, we'd be excited to have this in the core library. I'm happy to review a PR or two :-)

shounakpaul95 commented 3 years ago

Thanks @JohnGiorgi This works.