microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14k stars 1.81k forks source link

dashboard says no more trials but trials are still running. #5640

Closed busFred closed 1 year ago

busFred commented 1 year ago

Describe the issue:

image image

Environment:

Configuration: irrelevant

Log message:

 - dispatcher.log:

[2023-07-19 14:18:38] INFO (nni.tuner.tpe/MainThread) Using random seed 1020725736 [2023-07-19 14:18:38] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started

 - nnictl stdout and stderr:  (same for both stdout and stderr)

Experiment xfjd2uhs start: 2023-07-19 14:18:36.824787



<!--
Where can you find the log files:
LOG: https://github.com/microsoft/nni/blob/master/docs/en_US/Tutorial/HowToDebug.md#experiment-root-director
STDOUT/STDERR: https://nni.readthedocs.io/en/stable/reference/nnictl.html#nnictl-log-stdout
-->

**How to reproduce it?**:
busFred commented 1 year ago

my bad; status turns to done when all trials finished.