search

ja-thomas / OMLbots

13 stars 6 forks source link

Error runs #28

Open PhilippPro opened 7 years ago

PhilippPro commented 7 years ago

Are not uploaded on OpenML, are they? Kind of problematic...

DanielKuehn87 commented 7 years ago

Hm, annoying. But I feel like we still can handle it somehow.

  1. Runs fail because of ressource restrictions: As long our predictions for runtime/memory are accurate, this shouldn't be an issue.
  2. Runs fail due to invalid hyperparameter settings: We won't be able to learn this, but I think in most cases these restrictions are known upfront, so we can set the param.set accordingly.

Any other cases where the failed runs might be usefull?

PhilippPro commented 7 years ago

Ranger is still making a lot of problems/error runs we have to fix this.

berndbischl commented 7 years ago

failed runs SHOULD be uploaded if the failure is nontrivial and potentially interesting for other persons.

so dont upload: (eg)

upload

but feel free to discuss here, i am a bit unsure

PhilippPro commented 7 years ago

I think it is even not possible to upload failed runs to OpenML, is it? (see https://github.com/openml/openml-r/issues/331)

At the moment I am a bit unhappy with the OpenML solution, because of the accessibility of the results, maybe we should upload the results also to a local database directly.

berndbischl commented 7 years ago

I think it is even not possible to upload failed runs to OpenML, is it? (see openml/openml-r#331)

then talk to @giuseppec. for WEKA they have been uploading failed runs for years.

At the moment I am a bit unhappy with the OpenML solution, because of the accessibility of the results, maybe we should upload the results also to a local database directly.

i really would not do this, ONLY if ABSOLUTELY needed. how about you leave the results on disk in the cache for now? and write a simple collector?