search

openml / automlbenchmark

OpenML AutoML Benchmarking Framework
https://openml.github.io/automlbenchmark
MIT License
405 stars 134 forks source link

Incremental assessment? #343

Open microprediction opened 3 years ago

microprediction commented 3 years ago

Hi there. I noticed that you don't run benchmarks too often and I'm sympathetic to that. Just an idea, have you thought about incremental assessment via Elo ratings or some other more sophisticated updating scheme? I had in mind something similar to time-series elo ratings which are produced via a script like this. If that's of interest I could try to send a PR one of these months.

PGijsbers commented 3 years ago

We're very close to running the benchmark again :) But it's always good to look at ideas to make having up-to-date results sustainable. We haven't yet discussed anything like the scheme you propose, I think it's intriguing and worth a closer look. I personally don't have time to look into it right now, but I'll make sure to take it up with to other authors as well.