allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.61k stars 651 forks source link

HPO does not complete after timeout is reached #1114

Open d13g0 opened 1 year ago

d13g0 commented 1 year ago

Describe the bug

HPO does not complete after timeout is reached

To reproduce

Here is my code: https://gist.github.com/d13g0/76f3d092313d261308528297da21bbba

I am checking on timeouts and the number of active tasks in the while loop.

Also, once the optimization is completed (zero active tasks left...) shouldn't that trigger the completion of the HPO task itself?

Thanks.

Expected behaviour

The HPO task completes when the timeout is reached

Environment

jkhenning commented 1 year ago

Hi @d13g0, in order to use the time limit, you need to specify it using set_time_limit(), otherwise the feature will be inactive

jkhenning commented 1 year ago

I'm sorry, I missed the use of optimization_time_limit 🙂