microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.06k stars 1.82k forks source link

NNI not running anymore without error messages when CPU reached 100% once #5775

Open BitCalSaul opened 6 months ago

BitCalSaul commented 6 months ago

Describe the issue: Once the CPU utilization reached 100% once, even though NNI will finish the running trials but will not run the remaining trials.

Environment:

How to reproduce it?: You could run a task that consumes CPU resources across multiple trials simultaneously, and you will observe this issue.

I think this issue is as the same as this one #965 .