search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
14.01k stars 1.81k forks source link

why just few trails are succeeded and rest are failed #5393

Open Nafees-060 opened 1 year ago

Nafees-060 commented 1 year ago

Describe the issue: I am trying to optimize hyperparameters using NNI toolkit. Initially, it was stuck in the waiting state even for 24 hours. But after reinstallation then, I received all the trial jobs that Failed. However, when I ran today, it gave me 4 trials that succeeded and 6784 trials that failed (see the attached screenshot). Indeed, it is surprising as I did not change anything but its works for 4 trials now. However, it still does not work for 6784 trials. I am trying to identify the issue. Can you please help me to fix this issue?

Environment:

Configuration:

searchSpaceFile: search_space.yaml # Specify the Search Space file path useAnnotation: false # If it is true, searchSpaceFile will be ignore. default: false

trialCommand: python3.9 main.py # NOTE: change "python3" to "python" if you are using Windows trialCodeDirectory: . # Specify the Trial file path trialGpuNumber: 1 # Each trial needs 1 gpu trialConcurrency: 30 # Run 30 trials concurrently

maxExperimentDuration: 24h # Stop generating all trials after 24 hour maxTrialNumber: 1000 # Generate at most 1000 trials

tuner: # Configure the tuning algorithm name: TPE classArgs: # Algorithm specific arguments optimize_mode: maximize # maximize or minimize the needed metrics

trainingService: # Configure the training platform platform: local # Include local, remote, pai, etc. gpuIndices: 0, 1, 2 # The gpu-id 2 and 3 will be used useActiveGpu: true # Whether to use the gpu that has been used by other processes. maxTrialNumberPerGpu: 10 # Default: 1. Specify how many trials can share one GPU.`

 - Search space:

batch_size: _type: choice _value: [20, 40, 80, 120] lr: _type: choice _value: [0.001, 0.0001, 0.00001, 0.000001] hid_dim: _type: choice _value: [8, 16, 32, 64, 128, 256] epochs: _type: choice _value: [40, 60, 80, 100]

dropout_prob: _type: uniform _value: [0.1, 0.9]



**Log message**:
 - nnimanager.log (mentioning some part of log):
 - [2023-02-22 13:33:35] INFO (main) Start NNI manager
[2023-02-22 13:33:36] INFO (NNIDataStore) Datastore initialization done
[2023-02-22 13:33:36] INFO (RestServer) Starting REST server at port 8080, URL prefix: "/"
[2023-02-22 13:33:36] INFO (RestServer) REST server started.
[2023-02-22 13:33:36] INFO (NNIManager) Starting experiment: uhecj47z
[2023-02-22 13:33:36] INFO (NNIManager) Setup training service...
[2023-02-22 13:33:36] INFO (LocalTrainingService) Construct local machine training service.
[2023-02-22 13:33:36] INFO (NNIManager) Setup tuner...
[2023-02-22 13:33:36] INFO (NNIManager) Change NNIManager status from: INITIALIZED to: RUNNING
[2023-02-22 13:33:37] INFO (NNIManager) Add event listeners
[2023-02-22 13:33:37] INFO (LocalTrainingService) Run local machine training service.
[2023-02-22 13:33:37] INFO (NNIManager) NNIManager received command from dispatcher: ID, 
[2023-02-22 13:33:37] WARNING (GPUScheduler) gpu_metrics file does not exist!

[2023-02-22 13:33:42] INFO (NNIManager) submitTrialJob: form: {
  sequenceId: 0,
  hyperParameters: {
    value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 16, "epochs": 40, "dropout_prob": 0.2982458691630148}, "parameter_index": 0}',
    index: 0
  },
  placementConstraint: { type: 'None', gpus: [] }
}
[2023-02-22 13:34:03] INFO (NNIManager) Trial job KrG2B status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job jYhJQ status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job FEBWx status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job bTYBt status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job EbYU1 status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job wc9Dw status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job DrMCn status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job HQjDl status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job t3IWg status changed from RUNNING to FAILED
[2023-02-22 13:34:03] INFO (NNIManager) Trial job zOSAp status changed from RUNNING to FAILED

 - dispatcher.log:
 [2023-02-22 13:33:37] INFO (nni.tuner.tpe/MainThread) Using random seed 1844408314
[2023-02-22 13:33:37] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started

 - nnictl stdout and stderr:
View trail log:
{"error":"File not found: /home/anafees/nni-experiments/zofgdutp/trials/ZTHZg/trial.log"}
View trail Error:
{"error":"File not found: /home/anafees/nni-experiments/zofgdutp/trials/ZTHZg/stderr"}
View trail Stdout:
{"error":"File not found: /home/anafees/nni-experiments/zofgdutp/trials/ZTHZg/stdout"}

![1](https://user-images.githubusercontent.com/63704191/220535589-f31a5e31-dbb0-4d0c-b53e-167334f8d67c.png)
liuzhe-lz commented 1 year ago

Please check the files in nni-experiments/<EXPERIMENT-ID>/trials/<TRIAL-ID>. If they are empty, please run the experiment in debug mode and paste nnimanager.log.

Lijiaoa commented 1 year ago

Could you provide some message for debug? @Nafees-060

Nafees-060 commented 1 year ago

Could you provide some message for debug? @Nafees-060

Thank you for asking. Can you please tell me the exact command for debugging?

Lijiaoa commented 1 year ago

Please refer this doc

Nafees-060 commented 1 year ago

nni_manager.log in debugging: (its not complete)

[2023-03-08 17:00:17] INFO (main) Start NNI manager [2023-03-08 17:00:17] DEBUG (SqlDB) Database directory: /home/anafees/nni-experiments/km6h9avd/db [2023-03-08 17:00:17] INFO (NNIDataStore) Datastore initialization done [2023-03-08 17:00:17] INFO (RestServer) Starting REST server at port 8080, URL prefix: "/" [2023-03-08 17:00:17] INFO (RestServer) REST server started. [2023-03-08 17:00:17] DEBUG (main) start() returned. [2023-03-08 17:00:18] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2023-03-08 17:00:18] DEBUG (NNIRestHandler) POST: /experiment: body: { experimentName: 'HPO_real_world', experimentType: 'hpo', searchSpaceFile: '/home/anafees/HPO-HARProject4-FE/search_space.yaml', searchSpace: { batch_size: { _type: 'choice', _value: [Array] }, lr: { _type: 'choice', _value: [Array] }, hid_dim: { _type: 'choice', _value: [Array] }, channel_dim: { _type: 'choice', _value: [Array] }, time_reduce_size: { _type: 'choice', _value: [Array] }, epochs: { _type: 'choice', _value: [Array] }, dropout_prob: { _type: 'uniform', _value: [Array] } }, trialCommand: 'python3.9 main.py', trialCodeDirectory: '/home/anafees/HPO-HARProject4-FE', trialConcurrency: 30, trialGpuNumber: 3, maxExperimentDuration: '24h', maxTrialNumber: 2000, useAnnotation: false, debug: false, logLevel: 'info', experimentWorkingDirectory: '/home/anafees/nni-experiments', tuner: { name: 'TPE', classArgs: { optimize_mode: 'maximize' } }, trainingService: { platform: 'local', trialCommand: 'python3.9 main.py', trialCodeDirectory: '/home/anafees/HPO-HARProject4-FE', trialGpuNumber: 3, debug: false, useActiveGpu: true, maxTrialNumberPerGpu: 10, gpuIndices: [ 0, 1, 2 ], reuseMode: false } } [2023-03-08 17:00:18] INFO (NNIManager) Starting experiment: km6h9avd [2023-03-08 17:00:18] INFO (NNIManager) Setup training service... [2023-03-08 17:00:18] INFO (LocalTrainingService) Construct local machine training service. [2023-03-08 17:00:18] INFO (NNIManager) Setup tuner... [2023-03-08 17:00:18] DEBUG (NNIManager) dispatcher command: /usr/bin/python3.9,-m,nni,--exp_params,eyJleHBlcmltZW50TmFtZSI6IkhQT19yZWFsX3dvcmxkIiwiZXhwZXJpbWVudFR5cGUiOiJocG8iLCJzZWFyY2hTcGFjZUZpbGUiOiIvaG9tZS9hbmFmZWVzL0hQTy1IQVJQcm9qZWN0NC1GRS9zZWFyY2hfc3BhY2UueWFtbCIsInRyaWFsQ29tbWFuZCI6InB5dGhvbjMuOSBtYWluLnB5IiwidHJpYWxDb2RlRGlyZWN0b3J5IjoiL2hvbWUvYW5hZmVlcy9IUE8tSEFSUHJvamVjdDQtRkUiLCJ0cmlhbENvbmN1cnJlbmN5IjozMCwidHJpYWxHcHVOdW1iZXIiOjMsIm1heEV4cGVyaW1lbnREdXJhdGlvbiI6IjI0aCIsIm1heFRyaWFsTnVtYmVyIjoyMDAwLCJ1c2VBbm5vdGF0aW9uIjpmYWxzZSwiZGVidWciOmZhbHNlLCJsb2dMZXZlbCI6ImluZm8iLCJleHBlcmltZW50V29ya2luZ0RpcmVjdG9yeSI6Ii9ob21lL2FuYWZlZXMvbm5pLWV4cGVyaW1lbnRzIiwidHVuZXIiOnsibmFtZSI6IlRQRSIsImNsYXNzQXJncyI6eyJvcHRpbWl6ZV9tb2RlIjoibWF4aW1pemUifX0sInRyYWluaW5nU2VydmljZSI6eyJwbGF0Zm9ybSI6ImxvY2FsIiwidHJpYWxDb21tYW5kIjoicHl0aG9uMy45IG1haW4ucHkiLCJ0cmlhbENvZGVEaXJlY3RvcnkiOiIvaG9tZS9hbmFmZWVzL0hQTy1IQVJQcm9qZWN0NC1GRSIsInRyaWFsR3B1TnVtYmVyIjozLCJkZWJ1ZyI6ZmFsc2UsInVzZUFjdGl2ZUdwdSI6dHJ1ZSwibWF4VHJpYWxOdW1iZXJQZXJHcHUiOjEwLCJncHVJbmRpY2VzIjpbMCwxLDJdLCJyZXVzZU1vZGUiOmZhbHNlfX0= [2023-03-08 17:00:18] INFO (NNIManager) Change NNIManager status from: INITIALIZED to: RUNNING [2023-03-08 17:00:18] DEBUG (tuner_command_channel.WebSocketChannel) Waiting connection... [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Connected. [2023-03-08 17:00:19] INFO (NNIManager) Add event listeners [2023-03-08 17:00:19] DEBUG (NNIManager) Send tuner command: INITIALIZE: [object Object] [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Sending IN{"batch_size":{"_type":"choice","_value":[20,40,80,120]},"lr":{"_type":"choice","_value":[0.001,0.0001,0.00001,0.000001]},"hid_dim":{"_type":"choice","_value":[8,16,32,64,128,256]},"channel_dim":{"_type":"choice","_value":[8,16,32,64,128]},"time_reduce_size":{"_type":"choice","_value":[2,4,8,16,32]},"epochs":{"_type":"choice","_value":[40,60,80,100]},"dropout_prob":{"_type":"uniform","_value":[0.1,0.9]}} [2023-03-08 17:00:19] INFO (LocalTrainingService) Run local machine training service. [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received ID [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: ID, [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Sending GE30 [2023-03-08 17:00:19] WARNING (GPUScheduler) gpu_metrics file does not exist! [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8190542888283435}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8190542888283435}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.5652316403177816}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.5652316403177816}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5902149809150243}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5902149809150243}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.20486115983276562}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.20486115983276562}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.799097552291022}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.799097552291022}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.1813379412886767}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.1813379412886767}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.3028158037155387}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.3028158037155387}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.4328856227590965}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.4328856227590965}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 32, "epochs": 60, "dropout_prob": 0.7467162764382163}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 32, "epochs": 60, "dropout_prob": 0.7467162764382163}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 0.0001, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.6491025273180533}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 0.0001, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.6491025273180533}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.899055081378302}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.899055081378302}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 32, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.5671634248904714}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 32, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.5671634248904714}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.398005551675805}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.398005551675805}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 256, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7970531065421657}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 256, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7970531065421657}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 40, "dropout_prob": 0.49942346786997827}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 40, "dropout_prob": 0.49942346786997827}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 16, "time_reduce_size": 2, "epochs": 80, "dropout_prob": 0.6530039398681737}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 16, "time_reduce_size": 2, "epochs": 80, "dropout_prob": 0.6530039398681737}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.7095150163063442}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.7095150163063442}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7804439951900551}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7804439951900551}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.19220776868889164}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.19220776868889164}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4995960927309151}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4995960927309151}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.3075642518311735}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.3075642518311735}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8714931657592069}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8714931657592069}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 32, "epochs": 100, "dropout_prob": 0.6725181741501278}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 32, "epochs": 100, "dropout_prob": 0.6725181741501278}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.8518309067140063}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.8518309067140063}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5693901033194531}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5693901033194531}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 100, "dropout_prob": 0.41002242392277977}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 100, "dropout_prob": 0.41002242392277977}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.3155326764291908}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.3155326764291908}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.6036492009075237}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.6036492009075237}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 60, "dropout_prob": 0.7325897971832005}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 60, "dropout_prob": 0.7325897971832005}, "parameter_index": 0} [2023-03-08 17:00:19] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 128, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.8444230041648584}, "parameter_index": 0} [2023-03-08 17:00:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 128, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.8444230041648584}, "parameter_index": 0} [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 0, hyperParameters: { value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8190542888283435}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'wE9Lv', status: 'WAITING', submitTime: 1678266024226, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/wE9Lv', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/wE9Lv', form: { sequenceId: 0, hyperParameters: { value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8190542888283435}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 1, hyperParameters: { value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.5652316403177816}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 't1xft', status: 'WAITING', submitTime: 1678266024240, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/t1xft', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/t1xft', form: { sequenceId: 1, hyperParameters: { value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.5652316403177816}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 2, hyperParameters: { value: '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5902149809150243}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'ndiHy', status: 'WAITING', submitTime: 1678266024250, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/ndiHy', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/ndiHy', form: { sequenceId: 2, hyperParameters: { value: '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5902149809150243}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 3, hyperParameters: { value: '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.20486115983276562}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'S9f6X', status: 'WAITING', submitTime: 1678266024260, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/S9f6X', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/S9f6X', form: { sequenceId: 3, hyperParameters: { value: '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.20486115983276562}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 4, hyperParameters: { value: '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.799097552291022}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'psnoq', status: 'WAITING', submitTime: 1678266024269, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/psnoq', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/psnoq', form: { sequenceId: 4, hyperParameters: { value: '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.799097552291022}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 5, hyperParameters: { value: '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.1813379412886767}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'EiqQL', status: 'WAITING', submitTime: 1678266024278, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/EiqQL', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/EiqQL', form: { sequenceId: 5, hyperParameters: { value: '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.1813379412886767}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 6, hyperParameters: { value: '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.3028158037155387}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'fwrze', status: 'WAITING', submitTime: 1678266024289, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/fwrze', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/fwrze', form: { sequenceId: 6, hyperParameters: { value: '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.3028158037155387}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 7, hyperParameters: { value: '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.4328856227590965}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'VeLwM', status: 'WAITING', submitTime: 1678266024299, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/VeLwM', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/VeLwM', form: { sequenceId: 7, hyperParameters: { value: '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.4328856227590965}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 8, hyperParameters: { value: '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 32, "epochs": 60, "dropout_prob": 0.7467162764382163}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'b0LpU', status: 'WAITING', submitTime: 1678266024308, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/b0LpU', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/b0LpU', form: { sequenceId: 8, hyperParameters: { value: '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 32, "epochs": 60, "dropout_prob": 0.7467162764382163}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 9, hyperParameters: { value: '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 0.0001, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.6491025273180533}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'td7J0', status: 'WAITING', submitTime: 1678266024320, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/td7J0', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/td7J0', form: { sequenceId: 9, hyperParameters: { value: '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 0.0001, "hid_dim": 16, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.6491025273180533}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 10, hyperParameters: { value: '{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.899055081378302}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'LuYia', status: 'WAITING', submitTime: 1678266024330, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/LuYia', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/LuYia', form: { sequenceId: 10, hyperParameters: { value: '{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.899055081378302}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 11, hyperParameters: { value: '{"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 32, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.5671634248904714}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'T48rA', status: 'WAITING', submitTime: 1678266024341, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/T48rA', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/T48rA', form: { sequenceId: 11, hyperParameters: { value: '{"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 32, "channel_dim": 128, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.5671634248904714}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 12, hyperParameters: { value: '{"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.398005551675805}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'PdQvV', status: 'WAITING', submitTime: 1678266024351, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/PdQvV', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/PdQvV', form: { sequenceId: 12, hyperParameters: { value: '{"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.398005551675805}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 13, hyperParameters: { value: '{"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 256, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7970531065421657}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'LVM24', status: 'WAITING', submitTime: 1678266024361, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/LVM24', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/LVM24', form: { sequenceId: 13, hyperParameters: { value: '{"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 256, "channel_dim": 32, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7970531065421657}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 14, hyperParameters: { value: '{"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 40, "dropout_prob": 0.49942346786997827}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'gBzls', status: 'WAITING', submitTime: 1678266024371, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/gBzls', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/gBzls', form: { sequenceId: 14, hyperParameters: { value: '{"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 40, "dropout_prob": 0.49942346786997827}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 15, hyperParameters: { value: '{"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 16, "time_reduce_size": 2, "epochs": 80, "dropout_prob": 0.6530039398681737}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'iuZGt', status: 'WAITING', submitTime: 1678266024382, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/iuZGt', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/iuZGt', form: { sequenceId: 15, hyperParameters: { value: '{"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 16, "time_reduce_size": 2, "epochs": 80, "dropout_prob": 0.6530039398681737}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 16, hyperParameters: { value: '{"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.7095150163063442}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'z3amu', status: 'WAITING', submitTime: 1678266024391, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/z3amu', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/z3amu', form: { sequenceId: 16, hyperParameters: { value: '{"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 64, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.7095150163063442}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 17, hyperParameters: { value: '{"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7804439951900551}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'vJR34', status: 'WAITING', submitTime: 1678266024401, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/vJR34', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/vJR34', form: { sequenceId: 17, hyperParameters: { value: '{"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 8, "epochs": 60, "dropout_prob": 0.7804439951900551}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 18, hyperParameters: { value: '{"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.19220776868889164}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'GbX72', status: 'WAITING', submitTime: 1678266024412, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/GbX72', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/GbX72', form: { sequenceId: 18, hyperParameters: { value: '{"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.19220776868889164}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 19, hyperParameters: { value: '{"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4995960927309151}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'Ce10F', status: 'WAITING', submitTime: 1678266024422, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/Ce10F', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/Ce10F', form: { sequenceId: 19, hyperParameters: { value: '{"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4995960927309151}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 20, hyperParameters: { value: '{"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.3075642518311735}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'dqQRN', status: 'WAITING', submitTime: 1678266024434, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/dqQRN', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/dqQRN', form: { sequenceId: 20, hyperParameters: { value: '{"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.3075642518311735}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 21, hyperParameters: { value: '{"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8714931657592069}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'nfkeb', status: 'WAITING', submitTime: 1678266024445, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/nfkeb', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/nfkeb', form: { sequenceId: 21, hyperParameters: { value: '{"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.8714931657592069}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 22, hyperParameters: { value: '{"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 32, "epochs": 100, "dropout_prob": 0.6725181741501278}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'ynC3H', status: 'WAITING', submitTime: 1678266024456, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/ynC3H', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/ynC3H', form: { sequenceId: 22, hyperParameters: { value: '{"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 32, "epochs": 100, "dropout_prob": 0.6725181741501278}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 23, hyperParameters: { value: '{"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.8518309067140063}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'lciXl', status: 'WAITING', submitTime: 1678266024465, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/lciXl', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/lciXl', form: { sequenceId: 23, hyperParameters: { value: '{"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 40, "dropout_prob": 0.8518309067140063}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 24, hyperParameters: { value: '{"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5693901033194531}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'Hb0St', status: 'WAITING', submitTime: 1678266024476, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/Hb0St', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/Hb0St', form: { sequenceId: 24, hyperParameters: { value: '{"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 64, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.5693901033194531}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 25, hyperParameters: { value: '{"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 100, "dropout_prob": 0.41002242392277977}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'yrMFs', status: 'WAITING', submitTime: 1678266024486, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/yrMFs', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/yrMFs', form: { sequenceId: 25, hyperParameters: { value: '{"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 100, "dropout_prob": 0.41002242392277977}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 26, hyperParameters: { value: '{"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.3155326764291908}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'YQuCx', status: 'WAITING', submitTime: 1678266024495, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/YQuCx', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/YQuCx', form: { sequenceId: 26, hyperParameters: { value: '{"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 256, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.3155326764291908}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 27, hyperParameters: { value: '{"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.6036492009075237}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'LCBpo', status: 'WAITING', submitTime: 1678266024505, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/LCBpo', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/LCBpo', form: { sequenceId: 27, hyperParameters: { value: '{"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.0001, "hid_dim": 8, "channel_dim": 8, "time_reduce_size": 4, "epochs": 40, "dropout_prob": 0.6036492009075237}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 28, hyperParameters: { value: '{"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 60, "dropout_prob": 0.7325897971832005}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'MLMrY', status: 'WAITING', submitTime: 1678266024514, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/MLMrY', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/MLMrY', form: { sequenceId: 28, hyperParameters: { value: '{"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.0001, "hid_dim": 256, "channel_dim": 64, "time_reduce_size": 2, "epochs": 60, "dropout_prob": 0.7325897971832005}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:24] INFO (NNIManager) submitTrialJob: form: { sequenceId: 29, hyperParameters: { value: '{"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 128, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.8444230041648584}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:24] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'PeP3J', status: 'WAITING', submitTime: 1678266024526, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/PeP3J', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/PeP3J', form: { sequenceId: 29, hyperParameters: { value: '{"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 0.001, "hid_dim": 64, "channel_dim": 128, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.8444230041648584}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:29] INFO (NNIManager) Trial job wE9Lv status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job t1xft status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job ndiHy status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job S9f6X status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job psnoq status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job EiqQL status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job fwrze status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job VeLwM status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job b0LpU status changed from WAITING to RUNNING [2023-03-08 17:00:29] INFO (NNIManager) Trial job td7J0 status changed from WAITING to RUNNING [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: t1xft, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job t1xft status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"t1xft","event":"FAILED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 20, \"lr\": 0.0001, \"hid_dim\": 256, \"channel_dim\": 8, \"time_reduce_size\": 8, \"epochs\": 60, \"dropout_prob\": 0.5652316403177816}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: ndiHy, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job ndiHy status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"ndiHy","event":"FAILED","hyper_params":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 0.0001, \"hid_dim\": 128, \"channel_dim\": 32, \"time_reduce_size\": 4, \"epochs\": 40, \"dropout_prob\": 0.5902149809150243}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: S9f6X, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job S9f6X status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"S9f6X","event":"FAILED","hyper_params":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 40, \"lr\": 1e-06, \"hid_dim\": 32, \"channel_dim\": 32, \"time_reduce_size\": 8, \"epochs\": 60, \"dropout_prob\": 0.20486115983276562}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: psnoq, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job psnoq status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"psnoq","event":"FAILED","hyper_params":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 40, \"lr\": 1e-05, \"hid_dim\": 64, \"channel_dim\": 8, \"time_reduce_size\": 32, \"epochs\": 40, \"dropout_prob\": 0.799097552291022}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: EiqQL, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job EiqQL status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"EiqQL","event":"FAILED","hyper_params":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 120, \"lr\": 1e-06, \"hid_dim\": 16, \"channel_dim\": 128, \"time_reduce_size\": 4, \"epochs\": 60, \"dropout_prob\": 0.1813379412886767}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: fwrze, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job fwrze status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"fwrze","event":"FAILED","hyper_params":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 20, \"lr\": 1e-05, \"hid_dim\": 128, \"channel_dim\": 32, \"time_reduce_size\": 16, \"epochs\": 60, \"dropout_prob\": 0.3028158037155387}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: VeLwM, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job VeLwM status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"VeLwM","event":"FAILED","hyper_params":"{\"parameter_id\": 7, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 120, \"lr\": 1e-05, \"hid_dim\": 16, \"channel_dim\": 8, \"time_reduce_size\": 4, \"epochs\": 60, \"dropout_prob\": 0.4328856227590965}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: b0LpU, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job b0LpU status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"b0LpU","event":"FAILED","hyper_params":"{\"parameter_id\": 8, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 0.001, \"hid_dim\": 128, \"channel_dim\": 64, \"time_reduce_size\": 32, \"epochs\": 60, \"dropout_prob\": 0.7467162764382163}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (LocalTrainingService) trialJob status update: td7J0, FAILED [2023-03-08 17:00:34] INFO (NNIManager) Trial job td7J0 status changed from RUNNING to FAILED [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"td7J0","event":"FAILED","hyper_params":"{\"parameter_id\": 9, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 40, \"lr\": 0.0001, \"hid_dim\": 16, \"channel_dim\": 128, \"time_reduce_size\": 8, \"epochs\": 80, \"dropout_prob\": 0.6491025273180533}, \"parameter_index\": 0}"} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Sending GE9 [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 30, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8729539019725849}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 30, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8729539019725849}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 31, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8908411751352758}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 31, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8908411751352758}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 32, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.769477549781856}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 32, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.769477549781856}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 33, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.1074322163928666}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 33, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.1074322163928666}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 34, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8989804759292355}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 34, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8989804759292355}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 35, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 80, "dropout_prob": 0.8162475211972888}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 35, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 80, "dropout_prob": 0.8162475211972888}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 36, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.7283248681464033}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 36, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.7283248681464033}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 37, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.6119289031748583}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 37, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.6119289031748583}, "parameter_index": 0} [2023-03-08 17:00:34] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 38, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 32, "channel_dim": 16, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.5420894639822076}, "parameter_index": 0} [2023-03-08 17:00:34] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 38, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 32, "channel_dim": 16, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.5420894639822076}, "parameter_index": 0} [2023-03-08 17:00:39] INFO (NNIManager) Trial job LuYia status changed from WAITING to RUNNING [2023-03-08 17:00:39] INFO (NNIManager) Trial job T48rA status changed from WAITING to RUNNING [2023-03-08 17:00:39] INFO (NNIManager) Trial job PdQvV status changed from WAITING to RUNNING [2023-03-08 17:00:39] INFO (NNIManager) Trial job LVM24 status changed from WAITING to RUNNING [2023-03-08 17:00:39] INFO (NNIManager) Trial job gBzls status changed from WAITING to RUNNING [2023-03-08 17:00:40] INFO (NNIManager) Trial job iuZGt status changed from WAITING to RUNNING [2023-03-08 17:00:40] INFO (NNIManager) Trial job z3amu status changed from WAITING to RUNNING [2023-03-08 17:00:40] INFO (NNIManager) Trial job vJR34 status changed from WAITING to RUNNING [2023-03-08 17:00:40] INFO (NNIManager) Trial job GbX72 status changed from WAITING to RUNNING [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 30, hyperParameters: { value: '{"parameter_id": 30, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8729539019725849}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'ND2Qz', status: 'WAITING', submitTime: 1678266040117, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/ND2Qz', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/ND2Qz', form: { sequenceId: 30, hyperParameters: { value: '{"parameter_id": 30, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8729539019725849}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 31, hyperParameters: { value: '{"parameter_id": 31, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8908411751352758}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'H5gxE', status: 'WAITING', submitTime: 1678266040125, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/H5gxE', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/H5gxE', form: { sequenceId: 31, hyperParameters: { value: '{"parameter_id": 31, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 16, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8908411751352758}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 32, hyperParameters: { value: '{"parameter_id": 32, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.769477549781856}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'W7qfR', status: 'WAITING', submitTime: 1678266040134, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/W7qfR', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/W7qfR', form: { sequenceId: 32, hyperParameters: { value: '{"parameter_id": 32, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.769477549781856}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 33, hyperParameters: { value: '{"parameter_id": 33, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.1074322163928666}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'BrCXi', status: 'WAITING', submitTime: 1678266040141, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/BrCXi', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/BrCXi', form: { sequenceId: 33, hyperParameters: { value: '{"parameter_id": 33, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.1074322163928666}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 34, hyperParameters: { value: '{"parameter_id": 34, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8989804759292355}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'P8Gnb', status: 'WAITING', submitTime: 1678266040151, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/P8Gnb', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/P8Gnb', form: { sequenceId: 34, hyperParameters: { value: '{"parameter_id": 34, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.8989804759292355}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 35, hyperParameters: { value: '{"parameter_id": 35, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 80, "dropout_prob": 0.8162475211972888}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'vbbFs', status: 'WAITING', submitTime: 1678266040165, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/vbbFs', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/vbbFs', form: { sequenceId: 35, hyperParameters: { value: '{"parameter_id": 35, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 64, "channel_dim": 64, "time_reduce_size": 16, "epochs": 80, "dropout_prob": 0.8162475211972888}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 36, hyperParameters: { value: '{"parameter_id": 36, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.7283248681464033}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'YP0bt', status: 'WAITING', submitTime: 1678266040174, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/YP0bt', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/YP0bt', form: { sequenceId: 36, hyperParameters: { value: '{"parameter_id": 36, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 16, "channel_dim": 8, "time_reduce_size": 32, "epochs": 40, "dropout_prob": 0.7283248681464033}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 37, hyperParameters: { value: '{"parameter_id": 37, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.6119289031748583}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'eojpm', status: 'WAITING', submitTime: 1678266040182, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/eojpm', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/eojpm', form: { sequenceId: 37, hyperParameters: { value: '{"parameter_id": 37, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 128, "time_reduce_size": 4, "epochs": 80, "dropout_prob": 0.6119289031748583}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 38, hyperParameters: { value: '{"parameter_id": 38, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 32, "channel_dim": 16, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.5420894639822076}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-03-08 17:00:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'mV7sM', status: 'WAITING', submitTime: 1678266040191, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/home/anafees/nni-experiments/km6h9avd/trials/mV7sM', workingDirectory: '/home/anafees/nni-experiments/km6h9avd/trials/mV7sM', form: { sequenceId: 38, hyperParameters: { value: '{"parameter_id": 38, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 0.001, "hid_dim": 32, "channel_dim": 16, "time_reduce_size": 4, "epochs": 60, "dropout_prob": 0.5420894639822076}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: wE9Lv, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job wE9Lv status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"wE9Lv","event":"FAILED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 0.001, \"hid_dim\": 64, \"channel_dim\": 64, \"time_reduce_size\": 4, \"epochs\": 40, \"dropout_prob\": 0.8190542888283435}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: T48rA, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job T48rA status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"T48rA","event":"FAILED","hyper_params":"{\"parameter_id\": 11, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 20, \"lr\": 0.0001, \"hid_dim\": 32, \"channel_dim\": 128, \"time_reduce_size\": 8, \"epochs\": 80, \"dropout_prob\": 0.5671634248904714}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: PdQvV, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job PdQvV status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"PdQvV","event":"FAILED","hyper_params":"{\"parameter_id\": 12, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 40, \"lr\": 1e-05, \"hid_dim\": 128, \"channel_dim\": 16, \"time_reduce_size\": 8, \"epochs\": 100, \"dropout_prob\": 0.398005551675805}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: LVM24, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job LVM24 status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"LVM24","event":"FAILED","hyper_params":"{\"parameter_id\": 13, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 120, \"lr\": 1e-06, \"hid_dim\": 256, \"channel_dim\": 32, \"time_reduce_size\": 8, \"epochs\": 60, \"dropout_prob\": 0.7970531065421657}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: gBzls, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job gBzls status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"gBzls","event":"FAILED","hyper_params":"{\"parameter_id\": 14, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 20, \"lr\": 0.001, \"hid_dim\": 128, \"channel_dim\": 16, \"time_reduce_size\": 8, \"epochs\": 40, \"dropout_prob\": 0.49942346786997827}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: iuZGt, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job iuZGt status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"iuZGt","event":"FAILED","hyper_params":"{\"parameter_id\": 15, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 0.0001, \"hid_dim\": 256, \"channel_dim\": 16, \"time_reduce_size\": 2, \"epochs\": 80, \"dropout_prob\": 0.6530039398681737}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: z3amu, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job z3amu status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"z3amu","event":"FAILED","hyper_params":"{\"parameter_id\": 16, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 1e-05, \"hid_dim\": 128, \"channel_dim\": 64, \"time_reduce_size\": 8, \"epochs\": 100, \"dropout_prob\": 0.7095150163063442}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (LocalTrainingService) trialJob status update: vJR34, FAILED [2023-03-08 17:00:45] INFO (NNIManager) Trial job vJR34 status changed from RUNNING to FAILED [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"vJR34","event":"FAILED","hyper_params":"{\"parameter_id\": 17, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 0.0001, \"hid_dim\": 8, \"channel_dim\": 8, \"time_reduce_size\": 8, \"epochs\": 60, \"dropout_prob\": 0.7804439951900551}, \"parameter_index\": 0}"} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Sending GE8 [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 39, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.15518981291399891}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 39, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.15518981291399891}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 40, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.20747115012255743}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 40, "parameter_source": "algorithm", "parameters": {"batch_size": 120, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.20747115012255743}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 41, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.4415342147687348}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 41, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-05, "hid_dim": 128, "channel_dim": 16, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.4415342147687348}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 42, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.3127078324275976}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 42, "parameter_source": "algorithm", "parameters": {"batch_size": 40, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.3127078324275976}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 43, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.2438607073825832}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 43, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-06, "hid_dim": 128, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.2438607073825832}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 44, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 8, "channel_dim": 16, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.39920981278678436}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 44, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 8, "channel_dim": 16, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.39920981278678436}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 45, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4798069042731008}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 45, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 16, "epochs": 60, "dropout_prob": 0.4798069042731008}, "parameter_index": 0} [2023-03-08 17:00:45] DEBUG (tuner_command_channel.WebSocketChannel) Received TR{"parameter_id": 46, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.13795591332456056}, "parameter_index": 0} [2023-03-08 17:00:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 46, "parameter_source": "algorithm", "parameters": {"batch_size": 20, "lr": 1e-05, "hid_dim": 128, "channel_dim": 8, "time_reduce_size": 8, "epochs": 100, "dropout_prob": 0.13795591332456056}, "parameter_index": 0} [2023-03-08 17:00:50] DEBUG (LocalTrainingService) trialJob status update: LuYia, FAILED [2023-03-08 17:00:50] INFO (NNIManager) Trial job LuYia status changed from RUNNING to FAILED [2023-03-08 17:00:50] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"LuYia","event":"FAILED","hyper_params":"{\"parameter_id\": 10, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 20, \"lr\": 1e-06, \"hid_dim\": 128, \"channel_dim\": 8, \"time_reduce_size\": 4, \"epochs\": 80, \"dropout_prob\": 0.899055081378302}, \"parameter_index\": 0}"} [2023-03-08 17:00:50] DEBUG (LocalTrainingService) trialJob status update: GbX72, FAILED [2023-03-08 17:00:50] INFO (NNIManager) Trial job GbX72 status changed from RUNNING to FAILED [2023-03-08 17:00:50] DEBUG (tuner_command_channel.WebSocketChannel) Sending EN{"trial_job_id":"GbX72","event":"FAILED","hyper_params":"{\"parameter_id\": 18, \"parameter_source\": \"algorithm\", \"parameters\": {\"batch_size\": 80, \"lr\": 1e-06, \"hid_dim\": 32, \"channel_dim\": 32, \"time_reduce_size\": 8, \"epochs\": 80, \"dropout_prob\": 0.19220776868889164}, \"parameter_index\": 0}"} [2023-03-08 17:00:50] INFO (NNIManager) Trial job Ce10F status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job dqQRN status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job nfkeb status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job ynC3H status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job lciXl status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job Hb0St status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job yrMFs status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job YQuCx status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job LCBpo status changed from WAITING to RUNNING [2023-03-08 17:00:50] INFO (NNIManager) Trial job MLMrY status changed from WAITING to RUNNING [2023-03-08 17:00:50] DEBUG (tuner_command_channel.WebSocketChannel) Sending GE2 [2023-03-08 17:00:50] INFO (NNIManager) submitTrialJob: form: { sequenceId: 39, hyperParameters: { value: '{"parameter_id": 39, "parameter_source": "algorithm", "parameters": {"batch_size": 80, "lr": 1e-06, "hid_dim": 32, "channel_dim": 32, "time_reduce_size": 8, "epochs": 80, "dropout_prob": 0.15518981291399891}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }

Nafees-060 commented 1 year ago

Could you provide some message for debug? @Nafees-060

Hi @Lijiaoa I am sorry, but if you do not want to answer then why do you ask that upload the debug version?

liuzhe-lz commented 1 year ago

Sorry for slow response. The NNI manager log looks normal.

The cause of trials' failure should have been logged in ~/nni-experiments/EXPERIMENT-ID/trials/TRIAL-ID/stderr. These files should also be accessible from web portal, by clicking stderr buttons in trial details page.

Lijiaoa commented 1 year ago

@Nafees-060 Did you have any updates to report?