search

NCATSTranslator / Relay

Autonomous relay system for NCATS Biomedical Data Translator
MIT License
5 stars 24 forks source link

When scores are all 0, rank is always set to 2 #623

Closed maximusunc closed 1 week ago

maximusunc commented 6 months ago

What is the purpose of rank in the ARS results, and how is it calculated? I would expect them all to either be unique or to be 1?

MarkDWilliams commented 6 months ago

There are some odd parts in how tie-breaking "pushes down" ranks on tie. The behavior is all in scoring.py, but that code was written by Rosina's team and just ported over to the ARS. I'm open to changing how ties are handled.

maximusunc commented 6 months ago

Is rank used anywhere? UI? I was kind of assuming it was just the index of the result, but it sounds more complicated. Anyways, this came about because of the automated tests reports, so I made an issue there to not use rank in those reports: https://github.com/NCATSTranslator/ARS_Test_Runner/issues/30

MarkDWilliams commented 6 months ago

I'd have to double check with UI about whether they're using rank, but yeah it's more complicated than an index. I believe it was intended for some tie breaking behavior with weighted mean, though it doesn't work if the tie is the whole set

ShervinAbd92 commented 1 week ago

Gus just confirmed that UI team doesnt use the rank returned from ARS, they calculate rank on their end.