search

openml / OpenML

Open Machine Learning
https://openml.org
BSD 3-Clause "New" or "Revised" License
664 stars 90 forks source link

Add 'relative' evaluation measures #383

Open joaquinvanschoren opened 7 years ago

joaquinvanschoren commented 7 years ago
amueller commented 6 years ago

These can not be computed statically and therefore would need a whole new evaluation mechanism, right? The same prediction would have different scores every time a new run is uploaded (and the mean prediction changes). That would be a lot of re-computing evaluation measures.

joaquinvanschoren commented 6 years ago

We could compute them in the browser. We just need to download all results, compute the mean, then show relative results.

Through the API users would indeed need to compute this locally.

amueller commented 6 years ago

You just told me we can't even get all the results in the backend ;)

joaquinvanschoren commented 6 years ago

By default we indeed only fetch the first 5k :). And we can do it for all on demand as well.

On Sat, 17 Mar 2018 at 00:17 Andreas Mueller notifications@github.com wrote:

You just told me we can't even get all the results in the backend ;)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/openml/OpenML/issues/383#issuecomment-373869723, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV-HVQZBqlJSyppW85kKMEzoIwNJ5ks5tfEf3gaJpZM4MRQA8 .

-- Thank you, Joaquin