search

huggingface / evaluate

🤗 Evaluate: A library for easily evaluating machine learning models and datasets.
https://huggingface.co/docs/evaluate
Apache License 2.0
1.96k stars 249 forks source link

feature request: Dice metric for segmentation models #414

Open FlorinAndrei opened 1 year ago

FlorinAndrei commented 1 year ago

The library already has mean_iou (mean Jaccard index) which I have used for my image segmentation models and it works well.

If you look at recent research papers on the topic of image segmentation for medical imaging, you will notice that the Dice metric (Sorensen-Dice coefficient) is used a lot in research.

It would be very useful to have the Dice metric integrated into the main library - that would simplify model comparison when reviewing the literature. Thanks!

FlorinAndrei commented 1 year ago

There is a chance I may try to clone the IoU code and modify it to return Dice instead. The math is pretty similar. I do not know yet if I have the time to do it. I will know more in the next week or so. I'll post an update here.

sarthak247 commented 1 year ago

Greetings, I have been monitoring this issue for a while and it seems that there is still no one assigned to it or working on it. As someone who has previously worked with Segmentation I can understand how important Dice metric is in general and would like to know if someone's working on this yet. If not, is it fine if this gets assigned to me? I reviewed the code for mean_iou which was contributed by MMLLabs and feel that the same boilerplate can be used along with some changes to turn it into Dice Metric and shouldn't take long. Do let me know if someone's already working on it and if now I'm more than willing to take this up and start working on this ASAP <3

FlorinAndrei commented 1 year ago

@sarthak247 that sounds great! But I'm not a member of the Hugging Face team so I do not know if they are working on it or not.

Observation: it's important that the implementation is actually compliant with the definition of the Sorensen-Dice coefficient when tested on simple examples. I believe the current Hugging Face implementation of IoU (the Jaccard index), copied from OpenMMLab, does not pass simple tests, as shown here:

https://github.com/huggingface/evaluate/issues/421

https://github.com/open-mmlab/mmsegmentation/issues/2655