dispather comand：globals.args.pythonInterpreter, '-m', 'nni', '--exp_params', 占用内存太多

[yjjinjie]

Describe the issue:

Environment:

• NNI version: 2.9
• Training service (local|remote|pai|aml|etc): local
• Client OS:
• Server OS (for remote mode only):
• Python version:
• PyTorch/TensorFlow version:
• Is conda/virtualenv/venv used?:
• Is running in Docker?:

Configuration:

• Experiment config (remember to remove secrets!):

  assessor:
  classArgs:
  earlystop: true
  optimize_mode: maximize
  start_step: 5
  name: PAIAssessor
  experimentName: "\u65B0\u589E-deepfm-\u6FC0\u6D3B\u6A21\u578B\u4ED8\u8D39\u7387\u9884\
  \u4F30_0117_copy_copy_copy_copy_copy_copy_copy"
  experimentWorkingDirectory: ../expdir
  maxTrialNumber: 200
  searchSpaceFile: search_space.json
  trainingService:
  platform: local
  trialCommand: python3 -m hpo_tools.core.utils.run --config=./config.ini
  trialConcurrency: 2
  tuner:
  classArgs:
  optimize_mode: maximize
  name: TPE

• Search space:

  {"${batch_size}":{"_type":"choice","_value":["64","256","512","1024","5000","1500","2500","7500","10000"]},"${learning_rate}":{"_type":"choice","_value":["1e-5","1e-4","1e-3","5e-5","5e-4","5e-3"]},"${deep_dnn1_units}":{"_type":"randint","_value":[10,1000]},"${deep_dnn2_units}":{"_type":"randint","_value":[10,1000]},"${fm_dnn_units}":{"_type":"randint","_value":[10,1000]},"${combine_dnn_units}":{"_type":"randint","_value":[10,100]},"${seed}":{"_type":"randint","_value":[1,4294967290]},"${epochs}":{"_type":"randint","_value":[3,50]}}

Log message:

• nnimanager.log:

  placementConstraint: { type: 'None', gpus: [] }
  }
  [2024-01-24 15:37:33] INFO (NNIManager) Trial job zIKH1 status changed from WAITING to RUNNING
  [2024-01-24 15:49:24] INFO (NNIManager) Trial job ndLXi status changed from RUNNING to SUCCEEDED
  [2024-01-24 15:50:01] ERROR (tuner_command_channel.WebSocketChannel) Error: Error: tuner_command_channel: Tuner closed connection
  at WebSocket.<anonymous> (/usr/lib/python3.7/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:41:49)
  at WebSocket.emit (node:events:538:35)
  at WebSocket.emitClose (/usr/lib/python3.7/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:246:10)
  at Socket.socketOnClose (/usr/lib/python3.7/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:1127:15)
  at Socket.emit (node:events:526:28)
  at TCP.<anonymous> (node:net:687:12)
  [2024-01-24 15:50:01] ERROR (NNIManager) Dispatcher error: tuner_command_channel: Tuner closed connection
  [2024-01-24 15:50:01] ERROR (NNIManager) Error: Dispatcher stream error, tuner may have crashed.
  at EventEmitter.<anonymous> (/usr/lib/python3.7/site-packages/nni_node/core/nnimanager.js:647:32)
  at EventEmitter.emit (node:events:526:28)
  at WebSocketChannelImpl.handleError (/usr/lib/python3.7/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:107:22)
  at WebSocket.<anonymous> (/usr/lib/python3.7/site-packages/nni_node/core/tuner_command_channel/websocket_channel.js:41:37)
  at WebSocket.emit (node:events:538:35)
  at WebSocket.emitClose (/usr/lib/python3.7/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:246:10)
  at Socket.socketOnClose (/usr/lib/python3.7/site-packages/nni_node/node_modules/express-ws/node_modules/ws/lib/websocket.js:1127:15)
  at Socket.emit (node:events:526:28)
  at TCP.<anonymous> (node:net:687:12)
  [2024-01-24 15:50:01] INFO (NNIManager) Change NNIManager status from: RUNNING to: ERROR
  [2024-01-24 16:05:49] INFO (NNIManager) User cancelTrialJob: zIKH1
  [2024-01-24 16:05:49] INFO (ShutdownManager) Initiate shutdown: REST request
  [2024-01-24 16:05:49] INFO (RestServer) Stopping REST server.
  [2024-01-24 16:05:49] INFO (NNIManager) Change NNIManager status from: ERROR to: STOPPING
  [2024-01-24 16:05:49] INFO (NNIManager) Stopping experiment, cleaning up ...
  [2024-01-24 16:05:49] INFO (RestServer) REST server stopped.
  [2024-01-24 16:05:49] INFO (LocalTrainingService) Stopping local machine training service...
  [2024-01-24 16:05:49] INFO (NNIManager) Change NNIManager status from: STOPPING to: STOPPED
  [2024-01-24 16:05:49] INFO (NNIManager) Experiment stopped.
  [2024-01-24 16:05:49] INFO (NNITensorboardManager) Forced stopping all tensorboard task.
  [2024-01-24 16:05:49] INFO (NNITensorboardManager) All tensorboard task stopped.
  [2024-01-24 16:05:49] INFO (NNITensorboardManager) Tensorboard manager stopped.
  [2024-01-24 16:05:49] INFO (ShutdownManager) Shutdown complete.

• dispatcher.log: 没有报错
• nnictl stdout and stderr: 没有报错

How to reproduce it?:

[yjjinjie]

1）/python -m nni --exp_params 该进程会从2G->66G->98G;

2）出现时机在生成新参数的时候

3）发现是search space设置的空间太大

内存OOM："${seed}": {"_type": "randint","_value": [1, 4294967290] },

内存正常："${seed}": {"_type": "randint","_value": [1, 100] },

内存正常："${seed}": {"_type": "randint","_value": [4294967280, 4294967290] },