comic / grand-challenge.org

A platform for end-to-end development of machine learning solutions in biomedical imaging
https://grand-challenge.org
Apache License 2.0
168 stars 50 forks source link

Additional result metrics on leaderboards are not sortable #3276

Closed chrisvanrun closed 3 months ago

chrisvanrun commented 3 months ago

I would be quite curious to explore the results in ranking on these.

image

jmsmkn commented 3 months ago

Unfortunately this is a difficult problem to solve. The sorting is done in the db, there are too many entries to do this in the Django world or on the client side. We only index the main metric, not the others. I cannot see a way to do this that avoids having custom indices on the json object (which is pretty much unique per challenge) or a plethora of extra fields on the model. Do you have any other ideas?

pkcakeout commented 3 months ago

Stored functions in the db? :shudder:

But it is a solution that should be easy to implement at least!

chrisvanrun commented 3 months ago

Unfortunately this is a difficult problem to solve. The sorting is done in the db, there are too many entries to do this in the Django world or on the client side. We only index the main metric, not the others. I cannot see a way to do this that avoids having custom indices on the json object (which is pretty much unique per challenge) or a plethora of extra fields on the model. Do you have any other ideas?

I was afraid of that complication after first seeing it not being possible. The cost of user variability I guess. If you cannot readily think of a way to do this, I don't think I can readily suggest anything.

jmsmkn commented 3 months ago

I'm going to close this, would be nice to have but I cannot think of a straightforward way to do it.