search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.05k stars 1.82k forks source link

my experiment has error: ERROR (NNIManager) Dispatcher error: tuner_command_channel: Tuner closed connection #5006

Open nnnnnzy opened 2 years ago

nnnnnzy commented 2 years ago
e6775742c09794d18ca71dcf07cf8ee

nni-manager log file:

[2022-07-20 07:49:27] ERROR (tuner_command_channel.WebSocketChannel) Error: Error: tuner_command_channel: Tuner closed connection
    at WebSocket.<anonymous> (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:41:49)
    at WebSocket.emit (node:events:538:35)
    at WebSocket.emitClose (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:246:10)
    at Receiver.receiverOnFinish (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:1037:20)
    at Receiver.emit (node:events:526:28)
    at finish (node:internal/streams/writable:754:10)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
[2022-07-20 07:49:27] ERROR (NNIManager) Dispatcher error: tuner_command_channel: Tuner closed connection
[2022-07-20 07:49:27] ERROR (NNIManager) Error: Dispatcher stream error, tuner may have crashed.
    at EventEmitter.<anonymous> (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/core/nnimanager.js:647:32)
    at EventEmitter.emit (node:events:526:28)
    at WebSocketChannelImpl.handleError (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:107:22)
    at WebSocket.<anonymous> (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:41:37)
    at WebSocket.emit (node:events:538:35)
    at WebSocket.emitClose (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:246:10)
    at Receiver.receiverOnFinish (/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:1037:20)
    at Receiver.emit (node:events:526:28)
    at finish (node:internal/streams/writable:754:10)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
[2022-07-20 07:49:27] INFO (NNIManager) Change NNIManager status from: RUNNING to: ERROR
[2022-07-20 07:49:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:32] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:33] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:35] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:35] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:40] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:41] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:46] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:49] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:50] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:51] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:57] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:58] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:49:59] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:00] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:06] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:07] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:09] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:11] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:15] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:16] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:20] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:21] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:24] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:25] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:33] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:34] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:41] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:41] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:43] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:43] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:50] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:52] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:53] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:50:53] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:00] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:02] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:03] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:09] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:13] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:18] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:23] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:23] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:28] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:34] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:36] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:51:50] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:52:03] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:54:30] INFO (ShutdownManager) Initiate shutdown: REST request
[2022-07-20 07:54:30] INFO (RestServer) Stopping REST server.
[2022-07-20 07:54:30] INFO (NNIManager) Change NNIManager status from: ERROR to: STOPPING
[2022-07-20 07:54:30] INFO (NNIManager) Stopping experiment, cleaning up ...
[2022-07-20 07:54:30] WARNING (tuner_command_channel.WebSocketChannel) Sending too fast! Try to reduce the frequency of intermediate results.
[2022-07-20 07:54:30] INFO (RestServer) REST server stopped.
[2022-07-20 07:54:30] INFO (LocalTrainingService) Stopping local machine training service...
[2022-07-20 07:54:30] INFO (NNIManager) Change NNIManager status from: STOPPING to: STOPPED
[2022-07-20 07:54:30] INFO (NNIManager) Experiment stopped.
[2022-07-20 07:54:30] INFO (NNITensorboardManager) Forced stopping all tensorboard task.
[2022-07-20 07:54:30] INFO (NNITensorboardManager) All tensorboard task stopped.
[2022-07-20 07:54:30] INFO (NNITensorboardManager) Tensorboard manager stopped.
[2022-07-20 07:54:30] INFO (ShutdownManager) Shutdown complete.
[2022-07-20 08:07:12] INFO (main) Start NNI manager

nnimanager (1).log

dispatcher log file:


[2022-07-20 15:39:26] INFO (numexpr.utils/MainThread) Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2022-07-20 15:39:26] INFO (numexpr.utils/MainThread) Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2022-07-20 15:39:26] INFO (numexpr.utils/MainThread) NumExpr defaulting to 8 threads.
[2022-07-20 15:39:26] INFO (nni.tuner.random/MainThread) Using random seed 115296895
[2022-07-20 15:39:26] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started
[2022-07-20 15:48:14] ERROR (nni.main/MainThread) NNI manager closed connection
Traceback (most recent call last):
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/__main__.py", line 83, in <module>
    main()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/__main__.py", line 60, in main
    dispatcher.run()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/runtime/msg_dispatcher_base.py", line 68, in run
    command, data = self._channel._receive()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/runtime/tuner_command_channel/channel.py", line 59, in _receive
    raise RuntimeError('NNI manager closed connection')
RuntimeError: NNI manager closed connection
[2022-07-20 16:07:13] INFO (numexpr.utils/MainThread) Note: detected 255 virtual cores but NumExpr set to maximum of 64, check "NUMEXPR_MAX_THREADS" environment variable.
[2022-07-20 16:07:13] INFO (numexpr.utils/MainThread) Note: NumExpr detected 255 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
[2022-07-20 16:07:13] INFO (numexpr.utils/MainThread) NumExpr defaulting to 8 threads.
[2022-07-20 16:07:13] INFO (nni.tuner.random/MainThread) Using random seed 1509147283
[2022-07-20 16:07:13] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started
[2022-07-20 16:07:13] INFO (nni.tuner/MainThread) Load checkpoint ignored by tuner, checkpoint path: /home/nzy/nni-experiments/tgcfejsz/checkpoint
[2022-07-20 16:37:49] ERROR (nni.main/MainThread) NNI manager closed connection
Traceback (most recent call last):
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/__main__.py", line 83, in <module>
    main()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/__main__.py", line 60, in main
    dispatcher.run()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/runtime/msg_dispatcher_base.py", line 68, in run
    command, data = self._channel._receive()
  File "/home/nzy/miniconda3/envs/pyg1/lib/python3.8/site-packages/nni/runtime/tuner_command_channel/channel.py", line 59, in _receive
    raise RuntimeError('NNI manager closed connection')
RuntimeError: NNI manager closed connection

dispatcher (1).log

liuzhe-lz commented 2 years ago

How often does your trial code report intermediate result? If your epoch is short, you can try to report intermediate result per 10 epochs.

nnnnnzy commented 2 years ago

per 50 epochs.