Issues with DR/metrics/RMSE - Githubissues

openproblems-bio / openproblems

Formalizing and benchmarking open problems in single-cell genomics

MIT License

290 stars 77 forks source link

Issues with DR/metrics/RMSE #742

Closed scottgigante-immunai closed 1 year ago

scottgigante-immunai commented 1 year ago

[X] I have checked that this issue has not already been reported.
[X] I have confirmed this bug exists on the main branch of openproblems.

RMSE has two issues:

it is marked as maximise=True, meaning methods are rewarded for maximising RMSE. This seems like a bug.
It should probably be modified to be scale-invariant. If you multiply a DR by 10 it is no better or worse, but this metrics is penalizing methods that don't rescale their axes.