microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14k stars 1.81k forks source link

Dispatcher stream error, tuner may have crashed. #5724

Open QuintenDanneels opened 9 months ago

QuintenDanneels commented 9 months ago

Describe the issue:
I have been trying to apply GridSearch strategy to do some NAS experiments, but the tuner seems to crash constantly when starting an experiment. The python script works fine when I run it. When I look into the NNI manager log, it states that the Websocket is not open (readystate 3). So there is probably a problem with the connection, because it is closed by a peer. I already turned off my firewall and the problem still occurs.

Environment:

Configuration:

Log message: