Closed RobertTLange closed 2 years ago
Ideally, we would also have something similar for the case where an experiment fails due to something being wrong in the code. In this case the status would be set to failed
. In total there would be 5 experiment status: [running, completed, canceled, aborted, timed_out]
. This may be a little harder since we need a bullet proof way for detecting these failures. I am not sure how this would for example work out with the while-loop in the hyperparameter search which collects all logs. We may simply get stuck at this point?!
As of now experiments which where aborted/terminated before complication are listed as
running
in the status variable of thepickledb
instance. Given that the DB doesn't directly receive the info of the termination, it would be good to have some form of check/mechanism that sets the status toaborted
orcanceled
in this case.One potential way of doing so, would be via time constraints of experiments. For example if the user provides a time limit per job, we could calculate the maximum time for the entire experiment. If this time is exceeded and the status of the experiment is still
running
, we should change it. This could be checked everytime the db instance is loaded. The steps would look as follows:time_per_job: "dd:hh:mm"
given insingle_job_args
.total_experiment_time = num_search_batches * time_per_job
.max_completion_time = start_time + total_experiment_time
.current_time
>max_completion_time
andstatus == "running"
. If so - setstatus
toaborted
.