search

openml / openml-r

R package to interface with OpenML
http://openml.github.io/openml-r/
Other
95 stars 37 forks source link

Upload failed runs #371

Open DanielKuehn87 opened 7 years ago

DanielKuehn87 commented 7 years ago

Is it currently possible to upload failed runs using R? I'd like to know for which parameter-settings my mlr runs fail.

Seems like OpenML should be able to handle it: https://github.com/openml/OpenML/issues/424

joaquinvanschoren commented 7 years ago

Yes this is supported on the server On Fri, 23 Jun 2017 at 10:48, DanielKuehn87 notifications@github.com wrote:

Is it currently possible to upload failed runs using R? I'd like to know for which parameter-settings my mlr runs fail.

Seems like OpenML should be able to handle it: openml/OpenML#424 https://github.com/openml/OpenML/issues/424

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openml/openml-r/issues/371, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQVyDyLs-D8j2ZPk5rkRvHtgo_yq78ks5sG3v1gaJpZM4ODTKM .

-- Thank you, Joaquin

giuseppec commented 7 years ago

Unfortunately, the R package currently does not support this (at least this is not tested). Would you be willing to help here to start a PR?

joaquinvanschoren commented 7 years ago

Also, if you want to download failed runs, there is an option 'show_errors' that you must set to true, like this: https://www.openml.org/api/v1/run/list/uploader/86/show_errors/true

This is not documented ATM

On Fri, Jun 23, 2017 at 1:10 PM Giuseppe Casalicchio < notifications@github.com> wrote:

Unfortunately, the R package currently does not support this (at least this is not tested). Would you be willing to help here to start a PR?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/openml/openml-r/issues/371#issuecomment-310638833, or mute the thread https://github.com/notifications/unsubscribe-auth/ABpQV_doo4xmjLcdkuUkuoqFUW4lJ1ySks5sG50hgaJpZM4ODTKM .

-- Thank you, Joaquin

DanielKuehn87 commented 7 years ago

@joaquinvanschoren Where is the documentation for the upload of the failed runs? Is it just uploadrun without any predictions? https://www.openml.org/api_docs/#!/run/post_run I'll have a look, if I can manage to integrate it into the R interface.

giuseppec commented 7 years ago

Yeah, I think you need to make sure that runTaskMlr produces a mlr FailureModel object and try to upload the run without predictions (but using the error message of the FailureModel ).

joaquinvanschoren commented 7 years ago

I don't think this is explicitly documented, but the XSD shows how you can upload an error message with a run: https://www.openml.org/api/v1/xsd/openml.run.upload