search

CAMI-challenge / OPAL

OPAL: Open-community Profiling Assessment tooL
https://cami-challenge.github.io/OPAL/
Apache License 2.0
25 stars 6 forks source link

Scoring Ties #27

Closed nlapier2 closed 5 years ago

nlapier2 commented 5 years ago

Hello,

I really like the OPAL software! The automated scoring and visualizations are great. However, there's one behavior that seems a bit odd to me. It seems that ties are broken by method/label order, e.g. when two submitted methods are tied for a metric, the one listed first in the label order "wins". Is there a reason why ties are not just counted as ties? For instance, two methods could "tie" for best recall at the phylum level and both get the score 0, and then the next best method would have a score of 2.

Thanks, Nathan

dkoslicki commented 5 years ago

I completely agree: in the case of a tie, we should assign the same score to both.

I'm pretty sure this can be fixed here. @fernandomeyer I don't have a testing environment set up for OPAL, so do you want to fix it? Given enough time, I could probably do it, but I assume you're faster.

fernandomeyer commented 5 years ago

I agree too. I will look into this and will keep you updated. Thanks @nlapier2 and @dkoslicki for your feedback.

fernandomeyer commented 5 years ago

This is now fixed in commit 90b0bd5f399a8795392e56d5ec1b472fa38312b6 and version 1.0.1. Please reopen this issue should there still be a problem.