comet-ml / issue-tracking

Questions, Help, and Issues for Comet ML
https://www.comet.ml
85 stars 7 forks source link

Rerun + archive experiments from REST API #244

Closed neighthan closed 10 months ago

neighthan commented 5 years ago

(I've discussed this a bit on Slack, but I thought I'd make an official issue so that I can see when support for this comes; these are important features for me)

I often have several experiments fail for related reasons. I find it easier to analyze these failures in Python because I can easily aggregate by the error messages (possibly allowing for small differences, e.g. in whitespace). Once I resolve whatever errors caused the failures, I would like to be able to re-run the failed experiments and archive the previous run so that only the new (hopefully fixed) runs will show up. I would imagine this being implemented as an archive method on APIExperiment which does the same as clicking the archive button on the webpage for that experiment and, for allowing rerunning experiments, APIExperiment should have access to the information from the Reproduce section on the webpage (for this use case, only the Run command is needed, because one doesn't want to use the exact same code, but adding in the Reproduce section as well would be good since that should be easy to do at the same time and others could probably use that). This would improve my workflow a lot when I have hundreds of experiments in a project and need to sift through the errors to ensure I have all the results I need.

gidim commented 5 years ago

You can now Archive an experiment from the REST/Python API using this method:

import comet_ml
comet_api = comet_ml.API()
comet_api.archive_experiment("some key")` 
comet_api.archive_experiments(["key1","key2"])` 

Still working on the reproduce data

dsblank commented 5 years ago

@neighthan note that you'll need to upgrade your comet_ml to 1.0.50.

github-actions[bot] commented 10 months ago

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.