All trails are getting failed

[anandhperumal]

nni Environment:

• nni version: 0.9
• nni mode(local|pai|remote): local
• OS: windows
• python version: 3.6
• is conda or virtualenv used?: no
• is running in docker?: no

I'm just starting with NNI, I'm following the document step by step. And I see all my trails are getting failed

The exact command I ran and it's output:

C:\Users\User>nnictl create --config nni\examples\trials\mnist\config_windows.yml --port 8082
INFO: expand searchSpacePath: search_space.json to C:\Users\User\nni\examples\trials\mnist\search_space.json
INFO: expand codeDir: . to C:\Users\User\nni\examples\trials\mnist\.
INFO: Starting restful server...
INFO: Successfully started Restful server!
INFO: Setting local config...
INFO: Successfully set local config!
INFO: Starting experiment...
INFO: Successfully started experiment!
-----------------------------------------------------------------------
The experiment id is JAqxevyD
The Web UI urls are: http://169.254.170.118:8082   http://192.168.56.1:8082   http://169.254.221.8:8082   http://169.254.68.111:8082   http://10.0.0.215:8082   http://127.0.0.1:8082
-----------------------------------------------------------------------

You can use these commands to get more information about the experiment
-----------------------------------------------------------------------
         commands                       description
1. nnictl experiment show        show the information of experiments
2. nnictl trial ls               list all of trial jobs
3. nnictl top                    monitor the status of running experiments
4. nnictl log stderr             show stderr log content
5. nnictl log stdout             show stdout log content
6. nnictl stop                   stop an experiment
7. nnictl trial kill             kill a trial job by id
8. nnictl --help                 get help information about nnictl
-----------------------------------------------------------------------

I have attached the image of the WebUI which shows the trails has failed

Moreover, I keep getting a pop-up :

----------------------------------------------------------------
#!/usr/bin/env bash
# Copyright (C) 2014, Alexey Pavlov
#   mailto:alexpux@gmail.com
# This file is part of Minimal SYStem version 2.
#   https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$COMSPEC" "$@"
--------------------------------------------------------------------

how to disable it?

And where can I see logs? why are my trails failing? Please let me know if I'm missing anything. Any leads will be appreciated.

Thanks

[xuehui1991]

Hello @anandh02, you could use "nnictl experiment show JAqxevyD" to check the path of log.

You could check the stderr file in a trial folder and you could post the log of "dispatcher.log" and "nni_manager.log"

Thank you.

[anandhperumal]

Hi @xuehui1991, there is no message in stdrerr.

nnimanager logs:

---------------------------------------------------------------------------------------------
[7/25/2019, 1:29:51 AM] INFO [ 'Datastore initialization done' ]
[7/25/2019, 1:29:51 AM] INFO [ 'Rest server listening on: http://0.0.0.0:8082' ]
[7/25/2019, 1:29:51 AM] INFO [ 'RestServer start' ]
[7/25/2019, 1:29:51 AM] INFO [ 'Construct local machine training service.' ]
[7/25/2019, 1:29:51 AM] INFO [ 'RestServer base port is 8082' ]
[7/25/2019, 1:29:52 AM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python mnist.py","codeDir":"C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.","gpuNum":0}' ]
[7/25/2019, 1:29:52 AM] INFO [ 'required GPU number is 0' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Starting experiment: JAqxevyD' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Add event listeners' ]
[7/25/2019, 1:29:53 AM] INFO [ 'Run local machine training service.' ]
[7/25/2019, 1:29:54 AM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[7/25/2019, 1:29:54 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7430019065482085, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:29:58 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7430019065482085, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:30:08 AM] INFO [ 'Trial job JLCwt status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:08 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7055245865212583, "conv_size": 7, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:13 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7055245865212583, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:30:19 AM] INFO [ 'Trial job lRgva status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:19 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7104704507552245, "conv_size": 7, "hidden_size": 124, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:24 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7104704507552245, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:30:30 AM] INFO [ 'Trial job j5v2w status changed from WAITING to RUNNING' ]
[7/25/2019, 1:30:36 AM] INFO [ 'Trial job j5v2w status changed from RUNNING to FAILED' ]
[7/25/2019, 1:30:36 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6830578122412669, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:30:41 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6830578122412669, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:30:51 AM] INFO [ 'Trial job hyXdP status changed from WAITING to FAILED' ]
[7/25/2019, 1:30:51 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8794833854527577, "conv_size": 3, "hidden_size": 1024, "batch_size": 32, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/25/2019, 1:30:56 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8794833854527577, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 32, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:31:07 AM] INFO [ 'Trial job aJP2F status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:07 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6163793636385104, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:12 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6163793636385104, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:31:18 AM] INFO [ 'Trial job KhFdA status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:18 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7082566300398873, "conv_size": 5, "hidden_size": 1024, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/25/2019, 1:31:23 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7082566300398873, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:31:29 AM] INFO [ 'Trial job QHR8N status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:29 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8616380623467225, "conv_size": 5, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:34 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8616380623467225, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:31:45 AM] INFO [ 'Trial job i6is6 status changed from WAITING to FAILED' ]
[7/25/2019, 1:31:45 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8879214396569852, "conv_size": 3, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/25/2019, 1:31:50 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8879214396569852, \\"conv_size\\": 3, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:32:00 AM] INFO [ 'Trial job InrzT status changed from WAITING to FAILED' ]
[7/25/2019, 1:32:01 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.64958081561231, "conv_size": 3, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/25/2019, 1:32:06 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.64958081561231, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Trial job glM5m status changed from WAITING to FAILED' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Change NNIManager status from: RUNNING to: NO_MORE_TRIAL' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Change NNIManager status from: NO_MORE_TRIAL to: DONE' ]
[7/25/2019, 1:32:11 AM] INFO [ 'Experiment done.' ]
[7/25/2019, 1:32:11 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.53239802026354, "conv_size": 3, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}' ]
---------------------------------------------------------------------------------------------

dispatcher logs:

---------------------------------------------------------------------------------------------

[07/25/2019, 01:29:54 AM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/25/2019, 01:29:54 AM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/25/2019, 01:29:54 AM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/25/2019, 01:29:54 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005075 seconds
[07/25/2019, 01:29:54 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:08 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.001997 seconds
[07/25/2019, 01:30:08 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:19 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005175 seconds
[07/25/2019, 01:30:19 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:36 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003342 seconds
[07/25/2019, 01:30:36 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:30:51 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004096 seconds
[07/25/2019, 01:30:51 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:07 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.001040 seconds
[07/25/2019, 01:31:07 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:18 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002023 seconds
[07/25/2019, 01:31:18 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:29 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005700 seconds
[07/25/2019, 01:31:29 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:31:45 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005522 seconds
[07/25/2019, 01:31:45 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:32:01 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004645 seconds
[07/25/2019, 01:32:01 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/25/2019, 01:32:11 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005195 seconds
[07/25/2019, 01:32:11 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials

---------------------------------------------------------------------------------------------

[suiguoxin]

Pls check if tensorflow is installed. If yes, check stderr in experiments/JAqxevyD/trials/jLCwt/stderr

[xuehui1991]

From the "nni_manager" log, it seems has some error in trial code.

[anandhperumal]

@suiguoxin Yes, TensorFlow is installed and stderr file is empty

@xuehui1991 any suggestions? what could be the issue?

[suiguoxin]

@ult yuge zhang

[ultmaster]

Can you please give me all your contents in experiments/JAqxevyD/trials/jLCwt/?

Make sure you have the correct python selected and didn't mess around with Python 2 or Python 3 or whatever virtualenv you are using, which I assume you are not.

Also, try running the command after setting your working directory to C:\Users\User\nni\examples\trials\mnist\; that is, do nnictl create --config config_windows.yml straightly, instead of running it in C:\Users\User.

BTW, put your log in ```something``` (backticks) so that we can see clearly.

If it still doesn't work, feel free to ask.

[anandhperumal]

Hello @ultmaster

I'm using python 3 on local env. I tried changing the working directory to \mnsit\ still it's not working.

Inside experiments/JAqxevyD/trials/jLCwt/: parameter.cfg file contains : {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7430019065482085, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}

and run.ps1 file contains:

$env:NNI_PLATFORM="local"
$env:NNI_SYS_DIR="C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt"
$env:NNI_TRIAL_JOB_ID="JLCwt"
$env:NNI_OUTPUT_DIR="C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python mnist.py 2>C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt\.nni\state -NoNewline -encoding utf8 ```

stderr file is empty

and inside experiments\JAqxevyD\trials\JLCwt\.nni\
metrics file is empty
and state file contains :  1564039806036

Let me know if you wanna do a remote connection to look into the problem.
I'm sure, I'm missing something but not sure what is it.

[ultmaster]

Sorry for the delayed reply; it's Saturday here...

Are you sure it runs successfully when you directly run python mnist.py? When you are using nnictl, make sure it runs from the same console, as which you run python mnist.py previously.

Also you can try setting your working directory to where you put your mnist.py and call \the\path\to\your\run.ps1 and see what happens.

Furthermore, try creating a virtualenv and redo everything (re-install nni in virtualenv and etc.) If you don't know how to do it, Google it.

Remote connection would be great; but I'm not sure whether I can make it on time.

[anandhperumal]

Hi @ultmaster No problem.

I tried running python mnist.py (commented the NNI library calls in the code) It didn't run successfully.

C:\Users\User\nni\examples\trials\mnist>python mnist.py
WARNING:tensorflow:From mnist.py:151: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please write your own downloading logic.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\base.py:252: _internal_retry.<locals>.wrap.<locals>.wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
Instructions for updating:
Please use urllib or similar directly.
Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-images-idx3-ubyte.gz
Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.data to implement this functionality.
Extracting /tmp/tensorflow/mnist/input_data\train-labels-idx1-ubyte.gz
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:110: dense_to_one_hot (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use tf.one_hot on tensors.
Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
Extracting /tmp/tensorflow/mnist/input_data\t10k-images-idx3-ubyte.gz
Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
Extracting /tmp/tensorflow/mnist/input_data\t10k-labels-idx1-ubyte.gz
WARNING:tensorflow:From C:\Users\User\AppData\Roaming\Python\Python36\site-packages\tensorflow\contrib\learn\python\learn\datasets\mnist.py:290: DataSet.__init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
Instructions for updating:
Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
Mnist download data done.
WARNING:tensorflow:From mnist.py:113: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See `tf.nn.softmax_cross_entropy_with_logits_v2`.

2019-07-28 01:56:49.054102: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
2019-07-28 01:56:50.084176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1411] Found device 0 with properties:
name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.62
pciBusID: 0000:01:00.0
totalMemory: 4.00GiB freeMemory: 3.30GiB
2019-07-28 01:56:50.095127: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1490] Adding visible gpu devices: 0
2019-07-28 01:56:52.894762: I tensorflow/core/common_runtime/gpu/gpu_device.cc:971] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-07-28 01:56:52.899582: I tensorflow/core/common_runtime/gpu/gpu_device.cc:977]      0
2019-07-28 01:56:52.905277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0:   N
2019-07-28 01:56:52.910534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1103] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3013 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:01:00.0, compute capability: 6.1)
2019-07-28 01:56:55.704300: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.59GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:55.712542: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:55.719716: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.32GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:56.143649: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.90GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-07-28 01:56:56.154531: W tensorflow/core/common_runtime/bfc_allocator.cc:215] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.42GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.

It goes out of memory.

[ultmaster]

@anandh02 Set gpuNum to 0 in config_windows.yml and try again.

I'm pretty sure that "out of memory" is not the only problem here.

[anandhperumal]

@ultmaster gpuNum is 0 by default in config_windows.yml and any configuration in config_windows.yml shouldn't even matter I guess. Since I'm running >>python mnist.py

And I even tried running in virtualenv it fails over there too.

[ultmaster]

@anandh02 Re-read your log, and I believe that the error is not really "out of memory", it said that "The caller indicates that this is not a failure". So are you sure about the reason it fails? I mean, it would fail, since it couldn't get parameters from NNI.

Did you try nnictl in your virtualenv?

[anandhperumal]

@ultmaster I'm sorry, my bad. It ran successfully both in the local system and in virtualenv without NNI but through NNI it fails.

[ultmaster]

@anandh02 Okay, I just need to confirm everything here.

Of all the files listed in the screenshot above, you don't have trial.log and your stderr is empty. Am I right?

Can you try to run run.ps1 directly to see if everything is working fine?

BTW, please set --debug when you run nnictl and provide me with nnimanager.log and dispatcher.log one more time. thx

[anandhperumal]

@ultmaster Yes, I don't have trial.log and my stderr is empty.

nnimanager.logs

[7/29/2019, 12:47:21 AM] DEBUG [ 'Database directory: C:\\Users\\User\\nni\\experiments\\n5ssDtas\\db' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Datastore initialization done' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Rest server listening on: http://0.0.0.0:9056' ]
[7/29/2019, 12:47:21 AM] INFO [ 'RestServer start' ]
[7/29/2019, 12:47:21 AM] INFO [ 'Construct local machine training service.' ]
[7/29/2019, 12:47:21 AM] INFO [ 'RestServer base port is 9056' ]
[7/29/2019, 12:47:21 AM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 12:47:22 AM] DEBUG [ 'PUT: /experiment/cluster-metadata: body:\n{\n    "trial_config": {\n        "command": "python mnist.py",\n        "codeDir": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.",\n        "gpuNum": 0\n    }\n}' ]
[7/29/2019, 12:47:22 AM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python mnist.py","codeDir":"C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.","gpuNum":0}' ]
[7/29/2019, 12:47:22 AM] INFO [ 'required GPU number is 0' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'POST: /experiment: body:\n{\n    "authorName": "default",\n    "experimentName": "example_mnist",\n    "trialConcurrency": 1,\n    "maxExecDuration": 3600,\n    "maxTrialNum": 10,\n    "searchSpace": "{\\"dropout_rate\\": {\\"_type\\": \\"uniform\\", \\"_value\\": [0.5, 0.9]}, \\"conv_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [2, 3, 5, 7]}, \\"hidden_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [124, 512, 1024]}, \\"batch_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [1, 4, 8, 16, 32]}, \\"learning_rate\\": {\\"_type\\": \\"choice\\", \\"_value\\": [0.0001, 0.001, 0.01, 0.1]}}",\n    "trainingServicePlatform": "local",\n    "tuner": {\n        "builtinTunerName": "TPE",\n        "classArgs": {\n            "optimize_mode": "maximize"\n        },\n        "className": "TPE"\n    },\n    "versionCheck": false,\n    "clusterMetaData": [\n        {\n            "key": "codeDir",\n            "value": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\."\n        },\n        {\n            "key": "command",\n            "value": "python mnist.py"\n        }\n    ]\n}' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Starting experiment: n5ssDtas' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'Setup tuner...' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'dispatcher command: python -m nni --tuner_class_name TPE --tuner_args {"optimize_mode":"maximize"}' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Add event listeners' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'ipcInterface command type: [IN], content:[{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]' ]
[7/29/2019, 12:47:23 AM] INFO [ 'Run local machine training service.' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'Send tuner command: INITIALIZE: {"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}' ]
[7/29/2019, 12:47:23 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:24 AM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[7/29/2019, 12:47:24 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:24 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:28 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'submitTrialJob: return: {"id":"iz1GC","status":"WAITING","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:28 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"iz1GC","status":"WAITING","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[]}' ]
[7/29/2019, 12:47:33 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:38 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'trailJob status update: iz1GC, FAILED' ]
[7/29/2019, 12:47:39 AM] INFO [ 'Trial job iz1GC status changed from WAITING to FAILED' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"iz1GC","status":"FAILED","submitTime":1564382848578,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\iz1GC","sequenceId":0,"gpuIndices":[],"startTime":1564382855361,"pid":16016,"endTime":1564382859224}' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"iz1GC","event":"FAILED","hyper_params":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5040585530428365, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 8, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:47:39 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:39 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:47:43 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:44 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:44 AM] DEBUG [ 'submitTrialJob: return: {"id":"HGgpA","status":"WAITING","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:44 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"HGgpA","status":"WAITING","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[]}' ]
[7/29/2019, 12:47:48 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'trailJob status update: HGgpA, FAILED' ]
[7/29/2019, 12:47:49 AM] INFO [ 'Trial job HGgpA status changed from WAITING to FAILED' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"HGgpA","status":"FAILED","submitTime":1564382864238,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\HGgpA","sequenceId":1,"gpuIndices":[],"startTime":1564382867346,"pid":32020,"endTime":1564382869824}' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"HGgpA","event":"FAILED","hyper_params":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8993949196832431, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:47:49 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:47:49 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:47:53 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:47:54 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:47:54 AM] DEBUG [ 'submitTrialJob: return: {"id":"TjohE","status":"WAITING","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[]} ' ]
[7/29/2019, 12:47:54 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"TjohE","status":"WAITING","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[]}' ]
[7/29/2019, 12:47:58 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'trailJob status update: TjohE, FAILED' ]
[7/29/2019, 12:48:00 AM] INFO [ 'Trial job TjohE status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"TjohE","status":"FAILED","submitTime":1564382874839,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\TjohE","sequenceId":2,"gpuIndices":[],"startTime":1564382879241,"pid":36024,"endTime":1564382880419}' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"TjohE","event":"FAILED","hyper_params":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8654507247351698, \\"conv_size\\": 2, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:00 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:00 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 12:48:03 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:05 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:05 AM] DEBUG [ 'submitTrialJob: return: {"id":"knUSL","status":"WAITING","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:05 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"knUSL","status":"WAITING","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[]}' ]
[7/29/2019, 12:48:08 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:13 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'trailJob status update: knUSL, FAILED' ]
[7/29/2019, 12:48:16 AM] INFO [ 'Trial job knUSL status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"knUSL","status":"FAILED","submitTime":1564382885436,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\knUSL","sequenceId":3,"gpuIndices":[],"startTime":1564382891203,"pid":31676,"endTime":1564382896125}' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"knUSL","event":"FAILED","hyper_params":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7766423718362717, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 32, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:16 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:16 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 12:48:18 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:21 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:21 AM] DEBUG [ 'submitTrialJob: return: {"id":"n1HXB","status":"WAITING","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:21 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}, jobDetail: {"id":"n1HXB","status":"WAITING","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[]}' ]
[7/29/2019, 12:48:23 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'trailJob status update: n1HXB, FAILED' ]
[7/29/2019, 12:48:26 AM] INFO [ 'Trial job n1HXB status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"n1HXB","status":"FAILED","submitTime":1564382901141,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\n1HXB","sequenceId":4,"gpuIndices":[],"startTime":1564382903145,"pid":34220,"endTime":1564382906612}' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"n1HXB","event":"FAILED","hyper_params":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5424572727356373, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 16, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:26 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:26 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:28 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:31 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:31 AM] DEBUG [ 'submitTrialJob: return: {"id":"m0NKL","status":"WAITING","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:31 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"m0NKL","status":"WAITING","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[]}' ]
[7/29/2019, 12:48:33 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'trailJob status update: m0NKL, FAILED' ]
[7/29/2019, 12:48:37 AM] INFO [ 'Trial job m0NKL status changed from WAITING to FAILED' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"m0NKL","status":"FAILED","submitTime":1564382911624,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\m0NKL","sequenceId":5,"gpuIndices":[],"startTime":1564382915126,"pid":20080,"endTime":1564382917328}' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"m0NKL","event":"FAILED","hyper_params":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7972814466096125, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:37 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:37 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:38 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:42 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:42 AM] DEBUG [ 'submitTrialJob: return: {"id":"EdmO9","status":"WAITING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:42 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"EdmO9","status":"WAITING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[]}' ]
[7/29/2019, 12:48:43 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:47 AM] INFO [ 'Trial job EdmO9 status changed from WAITING to RUNNING' ]
[7/29/2019, 12:48:47 AM] DEBUG [ 'storeTrialJobEvent: event: RUNNING, data: undefined, jobDetail: {"id":"EdmO9","status":"RUNNING","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[],"startTime":1564382926952,"pid":27136}' ]
[7/29/2019, 12:48:48 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'trailJob status update: EdmO9, FAILED' ]
[7/29/2019, 12:48:53 AM] INFO [ 'Trial job EdmO9 status changed from RUNNING to FAILED' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"EdmO9","status":"FAILED","submitTime":1564382922342,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\EdmO9","sequenceId":6,"gpuIndices":[],"startTime":1564382926952,"pid":27136,"endTime":1564382933578}' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"EdmO9","event":"FAILED","hyper_params":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6800245119371473, \\"conv_size\\": 3, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:48:53 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:48:53 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:48:58 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'submitTrialJob: return: {"id":"jNswV","status":"WAITING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[]} ' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"jNswV","status":"WAITING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[]}' ]
[7/29/2019, 12:48:58 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:04 AM] INFO [ 'Trial job jNswV status changed from WAITING to RUNNING' ]
[7/29/2019, 12:49:04 AM] DEBUG [ 'storeTrialJobEvent: event: RUNNING, data: undefined, jobDetail: {"id":"jNswV","status":"RUNNING","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[],"startTime":1564382943513,"pid":41388}' ]
[7/29/2019, 12:49:04 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'trailJob status update: jNswV, FAILED' ]
[7/29/2019, 12:49:09 AM] INFO [ 'Trial job jNswV status changed from RUNNING to FAILED' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"jNswV","status":"FAILED","submitTime":1564382938592,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\jNswV","sequenceId":7,"gpuIndices":[],"startTime":1564382943513,"pid":41388,"endTime":1564382949785}' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"jNswV","event":"FAILED","hyper_params":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.529345593174789, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:09 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:09 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:14 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'submitTrialJob: return: {"id":"s1n9d","status":"WAITING","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[]} ' ]
[7/29/2019, 12:49:14 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"s1n9d","status":"WAITING","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[]}' ]
[7/29/2019, 12:49:19 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:24 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'trailJob status update: s1n9d, FAILED' ]
[7/29/2019, 12:49:25 AM] INFO [ 'Trial job s1n9d status changed from WAITING to FAILED' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"s1n9d","status":"FAILED","submitTime":1564382954795,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\s1n9d","sequenceId":8,"gpuIndices":[],"startTime":1564382960284,"pid":24072,"endTime":1564382965293}' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"s1n9d","event":"FAILED","hyper_params":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7487311518261832, \\"conv_size\\": 2, \\"hidden_size\\": 1024, \\"batch_size\\": 8, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:25 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:25 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 12:49:29 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:30 AM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 12:49:30 AM] DEBUG [ 'submitTrialJob: return: {"id":"XdGZ9","status":"WAITING","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[]} ' ]
[7/29/2019, 12:49:30 AM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"XdGZ9","status":"WAITING","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[]}' ]
[7/29/2019, 12:49:34 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:35 AM] INFO [ 'Change NNIManager status from: RUNNING to: NO_MORE_TRIAL' ]
[7/29/2019, 12:49:39 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'trailJob status update: XdGZ9, FAILED' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Trial job XdGZ9 status changed from WAITING to FAILED' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"XdGZ9","status":"FAILED","submitTime":1564382970310,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\n5ssDtas\\\\trials\\\\XdGZ9","sequenceId":9,"gpuIndices":[],"startTime":1564382977209,"pid":13720,"endTime":1564382981001}' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"XdGZ9","event":"FAILED","hyper_params":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7014019414695803, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 4, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Change NNIManager status from: NO_MORE_TRIAL to: DONE' ]
[7/29/2019, 12:49:41 AM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 12:49:41 AM] INFO [ 'Experiment done.' ]
[7/29/2019, 12:49:41 AM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040460409287476, "conv_size": 3, "hidden_size": 1024, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 12:49:44 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:49 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:54 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:49:59 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:04 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:09 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:14 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:19 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:24 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:29 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:34 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:39 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 12:50:44 AM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]

dispatcher.logs

[07/29/2019, 12:47:23 AM] DEBUG (nni.main/MainThread) START
[07/29/2019, 12:47:24 AM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 12:47:24 AM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher/MainThread) Assessor is not configured
[07/29/2019, 12:47:24 AM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'IN000322']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Initialize], data: [OrderedDict([('dropout_rate', OrderedDict([('_type', 'uniform'), ('_value', [0.5, 0.9])])), ('conv_size', OrderedDict([('_type', 'choice'), ('_value', [2, 3, 5, 7])])), ('hidden_size', OrderedDict([('_type', 'choice'), ('_value', [124, 512, 1024])])), ('batch_size', OrderedDict([('_type', 'choice'), ('_value', [1, 4, 8, 16, 32])])), ('learning_rate', OrderedDict([('_type', 'choice'), ('_value', [0.0001, 0.001, 0.01, 0.1])]))])]
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'ID000000']
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:24 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [0]
[07/29/2019, 12:47:24 AM] DEBUG (nni.tuner/Thread-1) generating param for 0
[07/29/2019, 12:47:24 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.000999 seconds
[07/29/2019, 12:47:24 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:24 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 12:47:28 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:28 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:28 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:33 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:33 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:33 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:38 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:38 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:38 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"iz1GC","event":"FAILED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.5040585530428365, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 8, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [1]
[07/29/2019, 12:47:39 AM] DEBUG (nni.tuner/Thread-1) generating param for 1
[07/29/2019, 12:47:39 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004124 seconds
[07/29/2019, 12:47:39 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'iz1GC'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040585530428365, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 12:47:39 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:47:43 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:43 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:43 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:48 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:48 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:48 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"HGgpA","event":"FAILED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8993949196832431, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 8, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'HGgpA'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8993949196832431, "conv_size": 2, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:47:49 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [2]
[07/29/2019, 12:47:49 AM] DEBUG (nni.tuner/Thread-1) generating param for 2
[07/29/2019, 12:47:49 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006025 seconds
[07/29/2019, 12:47:49 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:47:49 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:47:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:47:58 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:47:58 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:47:58 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"TjohE","event":"FAILED","hyper_params":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8654507247351698, \"conv_size\": 2, \"hidden_size\": 512, \"batch_size\": 16, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [3]
[07/29/2019, 12:48:00 AM] DEBUG (nni.tuner/Thread-1) generating param for 3
[07/29/2019, 12:48:00 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'TjohE'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8654507247351698, "conv_size": 2, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:00 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006348 seconds
[07/29/2019, 12:48:00 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:00 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 12:48:03 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:03 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:03 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:08 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:08 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:08 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:13 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:13 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:13 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000283']
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"knUSL","event":"FAILED","hyper_params":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7766423718362717, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 32, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'knUSL'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7766423718362717, "conv_size": 5, "hidden_size": 124, "batch_size": 32, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:16 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [4]
[07/29/2019, 12:48:16 AM] DEBUG (nni.tuner/Thread-1) generating param for 4
[07/29/2019, 12:48:16 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004289 seconds
[07/29/2019, 12:48:16 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:16 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 12:48:18 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:18 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:18 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:23 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:23 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:23 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"n1HXB","event":"FAILED","hyper_params":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.5424572727356373, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 16, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'n1HXB'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5424572727356373, "conv_size": 5, "hidden_size": 512, "batch_size": 16, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:26 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [5]
[07/29/2019, 12:48:26 AM] DEBUG (nni.tuner/Thread-1) generating param for 5
[07/29/2019, 12:48:26 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003028 seconds
[07/29/2019, 12:48:26 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:26 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:28 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:28 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:28 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:33 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:33 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:33 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"m0NKL","event":"FAILED","hyper_params":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7972814466096125, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [6]
[07/29/2019, 12:48:37 AM] DEBUG (nni.tuner/Thread-1) generating param for 6
[07/29/2019, 12:48:37 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'm0NKL'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7972814466096125, "conv_size": 7, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:37 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002901 seconds
[07/29/2019, 12:48:37 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:37 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:38 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:38 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:38 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:43 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:43 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:43 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:48 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:48 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:48 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000282']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"EdmO9","event":"FAILED","hyper_params":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6800245119371473, \"conv_size\": 3, \"hidden_size\": 1024, \"batch_size\": 16, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'EdmO9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6800245119371473, "conv_size": 3, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [7]
[07/29/2019, 12:48:53 AM] DEBUG (nni.tuner/Thread-1) generating param for 7
[07/29/2019, 12:48:53 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004421 seconds
[07/29/2019, 12:48:53 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000200{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:53 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:53 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:48:58 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:48:58 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:48:58 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000279']
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"jNswV","event":"FAILED","hyper_params":"{\"parameter_id\": 7, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.529345593174789, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'jNswV'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.529345593174789, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:09 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [8]
[07/29/2019, 12:49:09 AM] DEBUG (nni.tuner/Thread-1) generating param for 8
[07/29/2019, 12:49:09 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002992 seconds
[07/29/2019, 12:49:09 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:09 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:49:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:19 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:19 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:19 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"s1n9d","event":"FAILED","hyper_params":"{\"parameter_id\": 8, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7487311518261832, \"conv_size\": 2, \"hidden_size\": 1024, \"batch_size\": 8, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 's1n9d'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7487311518261832, "conv_size": 2, "hidden_size": 1024, "batch_size": 8, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:25 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [9]
[07/29/2019, 12:49:25 AM] DEBUG (nni.tuner/Thread-1) generating param for 9
[07/29/2019, 12:49:25 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004995 seconds
[07/29/2019, 12:49:25 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:25 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 12:49:29 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:29 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:29 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:34 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:34 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:34 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"XdGZ9","event":"FAILED","hyper_params":"{\"parameter_id\": 9, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7014019414695803, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 4, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'XdGZ9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7014019414695803, "conv_size": 5, "hidden_size": 512, "batch_size": 4, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 12:49:41 AM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [10]
[07/29/2019, 12:49:41 AM] DEBUG (nni.tuner/Thread-1) generating param for 10
[07/29/2019, 12:49:41 AM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005100 seconds
[07/29/2019, 12:49:41 AM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 12:49:41 AM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5040460409287476, "conv_size": 3, "hidden_size": 1024, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 12:49:44 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:44 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:44 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:54 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:54 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:54 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:49:59 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:49:59 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:49:59 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:19 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:19 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:19 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:24 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:24 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:24 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:29 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:29 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:29 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:34 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:34 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:34 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:39 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:39 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:39 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:44 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:44 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:44 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:49 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:49 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:49 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:54 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:54 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:54 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:50:59 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:50:59 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:50:59 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:04 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:04 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:04 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:09 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:09 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:09 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 12:51:14 AM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 12:51:14 AM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 12:51:14 AM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []

And how to run run.ps1

This is what comes when I try to run run.ps1 from cmd :

cd C:\Users\User\nni\examples\trials\mnist\.
$env:NNI_PLATFORM="local"
$env:NNI_SYS_DIR="C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC"
$env:NNI_TRIAL_JOB_ID="iz1GC"
$env:NNI_OUTPUT_DIR="C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python mnist.py 2>C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\.nni\state -NoNewline -encoding utf8

[ultmaster]

@anandh02 It seems that you didn't run it. Please do it in your Powershell:

.\run.ps1

Make sure your virtualenv is activated beforehand if you are using them. Then check your output in console and whatever is in C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr.

[anandhperumal]

@ultmaster By running run.ps1 This was the output

#!/usr/bin/env bash
# Copyright (C) 2014, Alexey Pavlov
#   mailto:alexpux@gmail.com
# This file is part of Minimal SYStem version 2.
#   https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$COMSPEC" "$@"

and the directory changed to nni\examples\trials\mnist

How come running run.ps1 will run the entire program? Since run.ps1 is in the folder nni\experiments\uH5ettE0\trials\ where \uH5ettE0 is a trial and it is generated after the running the program through nnictl.

Let me know if you're talking about some other run.ps1 file. Since I couldn't find anyother run.ps1 in my NNI directory.

[ultmaster]

@anandh02 If you tried to access run.ps1 with a text editor, you will find that it's a script hat does some environment settings, some cds and a python execution.

Now I cannot tell whether your python is successfully executed or not, since you didn't provide me with C:\Users\User\nni\experiments\n5ssDtas\trials\iz1GC\stderr. Please provide me with that info.

If it's convenient for you, I'm also interested in the exit code of .\run.ps1, you can get that by searching "print exit code of last command" on Google and do whatever works for you.

[anandhperumal]

@ultmaster

In run.ps1 there is no python execution command. It just sets some env variable and changes the dir and writes today's date in the file. To my understanding, any command (nnictl command) which will be running in cmd won't get affected because of this.

Still, I attached the screenshot of the execution of run.ps1

And if you look at the previous run.ps1 code I shared. There is already a command to print $LASTEXITCODE

And stderr is still empty.

Attaching the logs of nnimanager and dispatcher with debug mode.

dispatcher:

[07/29/2019, 10:04:33 PM] DEBUG (nni.main/MainThread) START
[07/29/2019, 10:04:34 PM] INFO (hyperopt.utils/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 10:04:34 PM] INFO (hyperopt.fmin/MainThread) Failed to load dill, try installing dill via "pip install dill" for enhanced pickling support.
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher/MainThread) Assessor is not configured
[07/29/2019, 10:04:34 PM] INFO (nni.msg_dispatcher_base/MainThread) Start dispatcher
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'IN000322']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Initialize], data: [OrderedDict([('dropout_rate', OrderedDict([('_type', 'uniform'), ('_value', [0.5, 0.9])])), ('conv_size', OrderedDict([('_type', 'choice'), ('_value', [2, 3, 5, 7])])), ('hidden_size', OrderedDict([('_type', 'choice'), ('_value', [124, 512, 1024])])), ('batch_size', OrderedDict([('_type', 'choice'), ('_value', [1, 4, 8, 16, 32])])), ('learning_rate', OrderedDict([('_type', 'choice'), ('_value', [0.0001, 0.001, 0.01, 0.1])]))])]
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'ID000000']
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:04:34 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [0]
[07/29/2019, 10:04:34 PM] DEBUG (nni.tuner/Thread-1) generating param for 0
[07/29/2019, 10:04:34 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.003999 seconds
[07/29/2019, 10:04:34 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:04:34 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}']
[07/29/2019, 10:04:37 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:37 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:37 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:42 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:42 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:42 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:47 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:47 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:47 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000283']
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"R1gZq","event":"FAILED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.8100726687947541, \"conv_size\": 5, \"hidden_size\": 512, \"batch_size\": 1, \"learning_rate\": 0.0001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'R1gZq'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}')])]
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [1]
[07/29/2019, 10:04:48 PM] DEBUG (nni.tuner/Thread-1) generating param for 1
[07/29/2019, 10:04:48 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.006125 seconds
[07/29/2019, 10:04:48 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:04:48 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 10:04:52 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:52 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:52 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:04:57 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:04:57 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:04:57 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:02 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:02 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:02 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"sZ8Br","event":"FAILED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.846059812817086, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [2]
[07/29/2019, 10:05:03 PM] DEBUG (nni.tuner/Thread-1) generating param for 2
[07/29/2019, 10:05:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'sZ8Br'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 10:05:03 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.010331 seconds
[07/29/2019, 10:05:03 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:03 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 10:05:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:12 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:12 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:12 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"WRRgp","event":"FAILED","hyper_params":"{\"parameter_id\": 2, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7355670839028239, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 8, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [3]
[07/29/2019, 10:05:14 PM] DEBUG (nni.tuner/Thread-1) generating param for 3
[07/29/2019, 10:05:14 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004090 seconds
[07/29/2019, 10:05:14 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:14 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 10:05:14 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'WRRgp'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 10:05:17 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:17 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:17 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:22 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:22 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:22 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"aKOrH","event":"FAILED","hyper_params":"{\"parameter_id\": 3, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6466162689528365, \"conv_size\": 5, \"hidden_size\": 124, \"batch_size\": 4, \"learning_rate\": 0.01}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [4]
[07/29/2019, 10:05:25 PM] DEBUG (nni.tuner/Thread-1) generating param for 4
[07/29/2019, 10:05:25 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004069 seconds
[07/29/2019, 10:05:25 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'aKOrH'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}')])]
[07/29/2019, 10:05:25 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:25 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:27 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:27 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:27 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:32 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:32 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:32 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:37 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:37 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:37 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"LRc4Z","event":"FAILED","hyper_params":"{\"parameter_id\": 4, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7149598193386633, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [5]
[07/29/2019, 10:05:40 PM] DEBUG (nni.tuner/Thread-1) generating param for 5
[07/29/2019, 10:05:40 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.005306 seconds
[07/29/2019, 10:05:40 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:40 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:40 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'LRc4Z'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:05:42 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:42 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:42 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:47 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:47 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:47 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:52 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:52 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:52 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"kDoG9","event":"FAILED","hyper_params":"{\"parameter_id\": 5, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6649355552943245, \"conv_size\": 2, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'kDoG9'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:05:56 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [6]
[07/29/2019, 10:05:56 PM] DEBUG (nni.tuner/Thread-1) generating param for 6
[07/29/2019, 10:05:56 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.002995 seconds
[07/29/2019, 10:05:56 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:05:56 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000201{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6894522330563883, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:05:57 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:05:57 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:05:57 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:02 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:02 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:02 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000280']
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"EOV8K","event":"FAILED","hyper_params":"{\"parameter_id\": 6, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6894522330563883, \"conv_size\": 7, \"hidden_size\": 124, \"batch_size\": 1, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [7]
[07/29/2019, 10:06:07 PM] DEBUG (nni.tuner/Thread-1) generating param for 7
[07/29/2019, 10:06:07 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004056 seconds
[07/29/2019, 10:06:07 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000202{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5621371754023703, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}']
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'EOV8K'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6894522330563883, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:07 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:07 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:12 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:12 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:12 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:18 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:18 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:18 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000281']
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"h68gu","event":"FAILED","hyper_params":"{\"parameter_id\": 7, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.5621371754023703, \"conv_size\": 7, \"hidden_size\": 512, \"batch_size\": 32, \"learning_rate\": 0.1}, \"parameter_index\": 0}"}]
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:06:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:06:23 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [8]
[07/29/2019, 10:06:23 PM] DEBUG (nni.tuner/Thread-1) generating param for 8
[07/29/2019, 10:06:23 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.004045 seconds
[07/29/2019, 10:06:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'h68gu'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5621371754023703, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}')])]
[07/29/2019, 10:06:23 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:06:23 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000205{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6546948554243907, "conv_size": 5, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}']
[07/29/2019, 10:06:28 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:28 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:28 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:33 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:33 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:33 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:38 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:38 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:38 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:39 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000284']
[07/29/2019, 10:06:39 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"sdWko","event":"FAILED","hyper_params":"{\"parameter_id\": 8, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.6546948554243907, \"conv_size\": 5, \"hidden_size\": 1024, \"batch_size\": 16, \"learning_rate\": 0.001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:06:39 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:06:39 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:06:39 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:06:39 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [9]
[07/29/2019, 10:06:39 PM] DEBUG (nni.tuner/Thread-1) generating param for 9
[07/29/2019, 10:06:39 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'sdWko'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6546948554243907, "conv_size": 5, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}')])]
[07/29/2019, 10:06:39 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.008047 seconds
[07/29/2019, 10:06:39 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:06:39 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000204{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7521816105091335, "conv_size": 7, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}']
[07/29/2019, 10:06:43 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:43 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:43 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:53 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:53 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:53 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:06:55 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'EN000283']
[07/29/2019, 10:06:55 PM] DEBUG (nni.protocol/MainThread) Received command, data: [{"trial_job_id":"OHFjd","event":"FAILED","hyper_params":"{\"parameter_id\": 9, \"parameter_source\": \"algorithm\", \"parameters\": {\"dropout_rate\": 0.7521816105091335, \"conv_size\": 7, \"hidden_size\": 512, \"batch_size\": 1, \"learning_rate\": 0.0001}, \"parameter_index\": 0}"}]
[07/29/2019, 10:06:55 PM] DEBUG (nni.msg_dispatcher_base/Thread-2) process_command: command: [CommandType.TrialEnd], data: [OrderedDict([('trial_job_id', 'OHFjd'), ('event', 'FAILED'), ('hyper_params', '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7521816105091335, "conv_size": 7, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}')])]
[07/29/2019, 10:06:55 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'GE000001']
[07/29/2019, 10:06:55 PM] DEBUG (nni.protocol/MainThread) Received command, data: [1]
[07/29/2019, 10:06:55 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.RequestTrialJobs], data: [1]
[07/29/2019, 10:06:55 PM] DEBUG (nni.msg_dispatcher/Thread-1) requesting for generating params of [10]
[07/29/2019, 10:06:55 PM] DEBUG (nni.tuner/Thread-1) generating param for 10
[07/29/2019, 10:06:55 PM] INFO (hyperopt.tpe/Thread-1) tpe_transform took 0.009106 seconds
[07/29/2019, 10:06:55 PM] INFO (hyperopt.tpe/Thread-1) TPE using 0 trials
[07/29/2019, 10:06:55 PM] DEBUG (nni.protocol/Thread-1) Sending command, data: [b'TR000203{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6052547440708993, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}']
[07/29/2019, 10:06:58 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:06:58 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:06:58 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:08 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:08 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:08 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:13 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:13 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:13 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:18 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:18 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:18 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:28 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:28 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:28 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:33 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:33 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:33 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:38 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:38 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:38 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:43 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:43 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:43 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:53 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:53 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:53 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:07:58 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:07:58 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:07:58 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:08 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:08 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:08 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:13 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:13 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:13 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:18 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:18 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:18 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:28 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:28 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:28 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:33 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:33 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:33 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:38 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:38 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:38 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:43 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:43 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:43 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:53 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:53 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:53 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:08:58 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:08:58 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:08:58 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:03 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:03 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:03 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:08 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:08 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:08 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:13 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:13 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:13 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:18 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:18 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:18 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:23 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:23 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:23 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:28 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:28 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:28 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:33 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:33 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:33 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:38 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:38 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:38 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:43 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:43 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:43 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:48 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:48 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:48 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []
[07/29/2019, 10:09:53 PM] DEBUG (nni.protocol/MainThread) Received command, header: [b'PI000000']
[07/29/2019, 10:09:53 PM] DEBUG (nni.protocol/MainThread) Received command, data: []
[07/29/2019, 10:09:53 PM] DEBUG (nni.msg_dispatcher_base/Thread-1) process_command: command: [CommandType.Ping], data: []

nnimanager:

[7/29/2019, 10:04:26 PM] DEBUG [ 'Database directory: C:\\Users\\User\\nni\\experiments\\tFYPOJX9\\db' ]
[7/29/2019, 10:04:26 PM] INFO [ 'Datastore initialization done' ]
[7/29/2019, 10:04:26 PM] INFO [ 'Rest server listening on: http://0.0.0.0:9052' ]
[7/29/2019, 10:04:26 PM] INFO [ 'RestServer start' ]
[7/29/2019, 10:04:26 PM] INFO [ 'Construct local machine training service.' ]
[7/29/2019, 10:04:26 PM] INFO [ 'RestServer base port is 9052' ]
[7/29/2019, 10:04:30 PM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 10:04:31 PM] DEBUG [ 'PUT: /experiment/cluster-metadata: body:\n{\n    "trial_config": {\n        "command": "python mnist.py",\n        "codeDir": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.",\n        "gpuNum": 0\n    }\n}' ]
[7/29/2019, 10:04:31 PM] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python mnist.py","codeDir":"C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\.","gpuNum":0}' ]
[7/29/2019, 10:04:31 PM] INFO [ 'required GPU number is 0' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'POST: /experiment: body:\n{\n    "authorName": "default",\n    "experimentName": "example_mnist",\n    "trialConcurrency": 1,\n    "maxExecDuration": 3600,\n    "maxTrialNum": 10,\n    "searchSpace": "{\\"dropout_rate\\": {\\"_type\\": \\"uniform\\", \\"_value\\": [0.5, 0.9]}, \\"conv_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [2, 3, 5, 7]}, \\"hidden_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [124, 512, 1024]}, \\"batch_size\\": {\\"_type\\": \\"choice\\", \\"_value\\": [1, 4, 8, 16, 32]}, \\"learning_rate\\": {\\"_type\\": \\"choice\\", \\"_value\\": [0.0001, 0.001, 0.01, 0.1]}}",\n    "trainingServicePlatform": "local",\n    "tuner": {\n        "builtinTunerName": "TPE",\n        "classArgs": {\n            "optimize_mode": "maximize"\n        },\n        "className": "TPE"\n    },\n    "versionCheck": false,\n    "clusterMetaData": [\n        {\n            "key": "codeDir",\n            "value": "C:\\\\Users\\\\User\\\\nni\\\\examples\\\\trials\\\\mnist\\\\."\n        },\n        {\n            "key": "command",\n            "value": "python mnist.py"\n        }\n    ]\n}' ]
[7/29/2019, 10:04:32 PM] INFO [ 'Starting experiment: tFYPOJX9' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'Setup tuner...' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'dispatcher command: python -m nni --tuner_class_name TPE --tuner_args {"optimize_mode":"maximize"}' ]
[7/29/2019, 10:04:32 PM] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[7/29/2019, 10:04:32 PM] INFO [ 'Add event listeners' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'ipcInterface command type: [IN], content:[{"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}]' ]
[7/29/2019, 10:04:32 PM] INFO [ 'Run local machine training service.' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'Send tuner command: INITIALIZE: {"dropout_rate": {"_type": "uniform", "_value": [0.5, 0.9]}, "conv_size": {"_type": "choice", "_value": [2, 3, 5, 7]}, "hidden_size": {"_type": "choice", "_value": [124, 512, 1024]}, "batch_size": {"_type": "choice", "_value": [1, 4, 8, 16, 32]}, "learning_rate": {"_type": "choice", "_value": [0.0001, 0.001, 0.01, 0.1]}}' ]
[7/29/2019, 10:04:32 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:04:34 PM] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[7/29/2019, 10:04:34 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:04:34 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/29/2019, 10:04:37 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:04:37 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8100726687947541, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:04:37 PM] DEBUG [ 'submitTrialJob: return: {"id":"R1gZq","status":"WAITING","submitTime":1564459477680,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8100726687947541, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","sequenceId":0,"gpuIndices":[]} ' ]
[7/29/2019, 10:04:37 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.8100726687947541, "conv_size": 5, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}, jobDetail: {"id":"R1gZq","status":"WAITING","submitTime":1564459477680,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8100726687947541, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","sequenceId":0,"gpuIndices":[]}' ]
[7/29/2019, 10:04:42 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:04:47 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:04:48 PM] DEBUG [ 'trailJob status update: R1gZq, FAILED' ]
[7/29/2019, 10:04:48 PM] INFO [ 'Trial job R1gZq status changed from WAITING to FAILED' ]
[7/29/2019, 10:04:48 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"R1gZq","status":"FAILED","submitTime":1564459477680,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8100726687947541, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\R1gZq","sequenceId":0,"gpuIndices":[],"startTime":1564459487365,"pid":21356,"endTime":1564459488160}' ]
[7/29/2019, 10:04:48 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"R1gZq","event":"FAILED","hyper_params":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.8100726687947541, \\"conv_size\\": 5, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:04:48 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:04:48 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 10:04:52 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:04:53 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.846059812817086, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:04:53 PM] DEBUG [ 'submitTrialJob: return: {"id":"sZ8Br","status":"WAITING","submitTime":1564459493180,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.846059812817086, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","sequenceId":1,"gpuIndices":[]} ' ]
[7/29/2019, 10:04:53 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.846059812817086, "conv_size": 2, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"sZ8Br","status":"WAITING","submitTime":1564459493180,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.846059812817086, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","sequenceId":1,"gpuIndices":[]}' ]
[7/29/2019, 10:04:57 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:02 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:03 PM] DEBUG [ 'trailJob status update: sZ8Br, FAILED' ]
[7/29/2019, 10:05:03 PM] INFO [ 'Trial job sZ8Br status changed from WAITING to FAILED' ]
[7/29/2019, 10:05:03 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"sZ8Br","status":"FAILED","submitTime":1564459493180,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.846059812817086, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sZ8Br","sequenceId":1,"gpuIndices":[],"startTime":1564459499391,"pid":10340,"endTime":1564459503861}' ]
[7/29/2019, 10:05:03 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"sZ8Br","event":"FAILED","hyper_params":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.846059812817086, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:05:03 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:05:03 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 10:05:07 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:08 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7355670839028239, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:05:08 PM] DEBUG [ 'submitTrialJob: return: {"id":"WRRgp","status":"WAITING","submitTime":1564459508880,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7355670839028239, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","sequenceId":2,"gpuIndices":[]} ' ]
[7/29/2019, 10:05:08 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7355670839028239, "conv_size": 5, "hidden_size": 124, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}, jobDetail: {"id":"WRRgp","status":"WAITING","submitTime":1564459508880,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7355670839028239, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","sequenceId":2,"gpuIndices":[]}' ]
[7/29/2019, 10:05:12 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:14 PM] DEBUG [ 'trailJob status update: WRRgp, FAILED' ]
[7/29/2019, 10:05:14 PM] INFO [ 'Trial job WRRgp status changed from WAITING to FAILED' ]
[7/29/2019, 10:05:14 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"WRRgp","status":"FAILED","submitTime":1564459508880,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7355670839028239, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\WRRgp","sequenceId":2,"gpuIndices":[],"startTime":1564459511551,"pid":10296,"endTime":1564459514569}' ]
[7/29/2019, 10:05:14 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"WRRgp","event":"FAILED","hyper_params":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7355670839028239, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 8, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:05:14 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:05:14 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 10:05:17 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:19 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6466162689528365, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:05:19 PM] DEBUG [ 'submitTrialJob: return: {"id":"aKOrH","status":"WAITING","submitTime":1564459519593,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6466162689528365, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","sequenceId":3,"gpuIndices":[]} ' ]
[7/29/2019, 10:05:19 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6466162689528365, "conv_size": 5, "hidden_size": 124, "batch_size": 4, "learning_rate": 0.01}, "parameter_index": 0}, jobDetail: {"id":"aKOrH","status":"WAITING","submitTime":1564459519593,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6466162689528365, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","sequenceId":3,"gpuIndices":[]}' ]
[7/29/2019, 10:05:22 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:25 PM] DEBUG [ 'trailJob status update: aKOrH, FAILED' ]
[7/29/2019, 10:05:25 PM] INFO [ 'Trial job aKOrH status changed from WAITING to FAILED' ]
[7/29/2019, 10:05:25 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"aKOrH","status":"FAILED","submitTime":1564459519593,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6466162689528365, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\aKOrH","sequenceId":3,"gpuIndices":[],"startTime":1564459523673,"pid":11816,"endTime":1564459525284}' ]
[7/29/2019, 10:05:25 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"aKOrH","event":"FAILED","hyper_params":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6466162689528365, \\"conv_size\\": 5, \\"hidden_size\\": 124, \\"batch_size\\": 4, \\"learning_rate\\": 0.01}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:05:25 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:05:25 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 10:05:27 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:30 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7149598193386633, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:05:30 PM] DEBUG [ 'submitTrialJob: return: {"id":"LRc4Z","status":"WAITING","submitTime":1564459530312,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7149598193386633, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","sequenceId":4,"gpuIndices":[]} ' ]
[7/29/2019, 10:05:30 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7149598193386633, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"LRc4Z","status":"WAITING","submitTime":1564459530312,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7149598193386633, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","sequenceId":4,"gpuIndices":[]}' ]
[7/29/2019, 10:05:32 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:37 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:40 PM] DEBUG [ 'trailJob status update: LRc4Z, FAILED' ]
[7/29/2019, 10:05:40 PM] INFO [ 'Trial job LRc4Z status changed from WAITING to FAILED' ]
[7/29/2019, 10:05:40 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"LRc4Z","status":"FAILED","submitTime":1564459530312,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7149598193386633, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\LRc4Z","sequenceId":4,"gpuIndices":[],"startTime":1564459535826,"pid":17244,"endTime":1564459540947}' ]
[7/29/2019, 10:05:40 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"LRc4Z","event":"FAILED","hyper_params":"{\\"parameter_id\\": 4, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7149598193386633, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:05:40 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:05:40 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 10:05:42 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:45 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6649355552943245, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:05:45 PM] DEBUG [ 'submitTrialJob: return: {"id":"kDoG9","status":"WAITING","submitTime":1564459545967,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6649355552943245, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","sequenceId":5,"gpuIndices":[]} ' ]
[7/29/2019, 10:05:46 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6649355552943245, "conv_size": 2, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"kDoG9","status":"WAITING","submitTime":1564459545967,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6649355552943245, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","sequenceId":5,"gpuIndices":[]}' ]
[7/29/2019, 10:05:47 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:52 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:05:56 PM] DEBUG [ 'trailJob status update: kDoG9, FAILED' ]
[7/29/2019, 10:05:56 PM] INFO [ 'Trial job kDoG9 status changed from WAITING to FAILED' ]
[7/29/2019, 10:05:56 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"kDoG9","status":"FAILED","submitTime":1564459545967,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6649355552943245, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\kDoG9","sequenceId":5,"gpuIndices":[],"startTime":1564459553066,"pid":32512,"endTime":1564459556724}' ]
[7/29/2019, 10:05:56 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"kDoG9","event":"FAILED","hyper_params":"{\\"parameter_id\\": 5, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6649355552943245, \\"conv_size\\": 2, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:05:56 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:05:56 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6894522330563883, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 10:05:57 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:01 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6894522330563883, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:06:01 PM] DEBUG [ 'submitTrialJob: return: {"id":"EOV8K","status":"WAITING","submitTime":1564459561749,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6894522330563883, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","sequenceId":6,"gpuIndices":[]} ' ]
[7/29/2019, 10:06:01 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6894522330563883, "conv_size": 7, "hidden_size": 124, "batch_size": 1, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"EOV8K","status":"WAITING","submitTime":1564459561749,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6894522330563883, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","sequenceId":6,"gpuIndices":[]}' ]
[7/29/2019, 10:06:02 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:07 PM] DEBUG [ 'trailJob status update: EOV8K, FAILED' ]
[7/29/2019, 10:06:07 PM] INFO [ 'Trial job EOV8K status changed from WAITING to FAILED' ]
[7/29/2019, 10:06:07 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"EOV8K","status":"FAILED","submitTime":1564459561749,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6894522330563883, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\EOV8K","sequenceId":6,"gpuIndices":[],"startTime":1564459565175,"pid":17112,"endTime":1564459567245}' ]
[7/29/2019, 10:06:07 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"EOV8K","event":"FAILED","hyper_params":"{\\"parameter_id\\": 6, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6894522330563883, \\"conv_size\\": 7, \\"hidden_size\\": 124, \\"batch_size\\": 1, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:06:07 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:06:07 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5621371754023703, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}' ]
[7/29/2019, 10:06:07 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:12 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:06:12 PM] DEBUG [ 'submitTrialJob: return: {"id":"h68gu","status":"WAITING","submitTime":1564459572376,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","sequenceId":7,"gpuIndices":[]} ' ]
[7/29/2019, 10:06:12 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.5621371754023703, "conv_size": 7, "hidden_size": 512, "batch_size": 32, "learning_rate": 0.1}, "parameter_index": 0}, jobDetail: {"id":"h68gu","status":"WAITING","submitTime":1564459572376,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","sequenceId":7,"gpuIndices":[]}' ]
[7/29/2019, 10:06:12 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:18 PM] INFO [ 'Trial job h68gu status changed from WAITING to RUNNING' ]
[7/29/2019, 10:06:18 PM] DEBUG [ 'storeTrialJobEvent: event: RUNNING, data: undefined, jobDetail: {"id":"h68gu","status":"RUNNING","submitTime":1564459572376,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","sequenceId":7,"gpuIndices":[],"startTime":1564459577332,"pid":416}' ]
[7/29/2019, 10:06:18 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:23 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:23 PM] DEBUG [ 'trailJob status update: h68gu, FAILED' ]
[7/29/2019, 10:06:23 PM] INFO [ 'Trial job h68gu status changed from RUNNING to FAILED' ]
[7/29/2019, 10:06:23 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"h68gu","status":"FAILED","submitTime":1564459572376,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\h68gu","sequenceId":7,"gpuIndices":[],"startTime":1564459577332,"pid":416,"endTime":1564459583897}' ]
[7/29/2019, 10:06:23 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"h68gu","event":"FAILED","hyper_params":"{\\"parameter_id\\": 7, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.5621371754023703, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 32, \\"learning_rate\\": 0.1}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:06:23 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:06:23 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6546948554243907, "conv_size": 5, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}' ]
[7/29/2019, 10:06:28 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:28 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6546948554243907, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:06:28 PM] DEBUG [ 'submitTrialJob: return: {"id":"sdWko","status":"WAITING","submitTime":1564459588917,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6546948554243907, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","sequenceId":8,"gpuIndices":[]} ' ]
[7/29/2019, 10:06:28 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6546948554243907, "conv_size": 5, "hidden_size": 1024, "batch_size": 16, "learning_rate": 0.001}, "parameter_index": 0}, jobDetail: {"id":"sdWko","status":"WAITING","submitTime":1564459588917,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6546948554243907, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","sequenceId":8,"gpuIndices":[]}' ]
[7/29/2019, 10:06:33 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:38 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:39 PM] DEBUG [ 'trailJob status update: sdWko, FAILED' ]
[7/29/2019, 10:06:39 PM] INFO [ 'Trial job sdWko status changed from WAITING to FAILED' ]
[7/29/2019, 10:06:39 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"sdWko","status":"FAILED","submitTime":1564459588917,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6546948554243907, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\sdWko","sequenceId":8,"gpuIndices":[],"startTime":1564459594424,"pid":12520,"endTime":1564459599632}' ]
[7/29/2019, 10:06:39 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"sdWko","event":"FAILED","hyper_params":"{\\"parameter_id\\": 8, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.6546948554243907, \\"conv_size\\": 5, \\"hidden_size\\": 1024, \\"batch_size\\": 16, \\"learning_rate\\": 0.001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:06:39 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:06:39 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7521816105091335, "conv_size": 7, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}' ]
[7/29/2019, 10:06:43 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:44 PM] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7521816105091335, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}}' ]
[7/29/2019, 10:06:44 PM] DEBUG [ 'submitTrialJob: return: {"id":"OHFjd","status":"WAITING","submitTime":1564459604651,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7521816105091335, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","sequenceId":9,"gpuIndices":[]} ' ]
[7/29/2019, 10:06:44 PM] DEBUG [ 'storeTrialJobEvent: event: WAITING, data: {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.7521816105091335, "conv_size": 7, "hidden_size": 512, "batch_size": 1, "learning_rate": 0.0001}, "parameter_index": 0}, jobDetail: {"id":"OHFjd","status":"WAITING","submitTime":1564459604651,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7521816105091335, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","sequenceId":9,"gpuIndices":[]}' ]
[7/29/2019, 10:06:48 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:49 PM] INFO [ 'Change NNIManager status from: RUNNING to: NO_MORE_TRIAL' ]
[7/29/2019, 10:06:53 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:06:55 PM] DEBUG [ 'trailJob status update: OHFjd, FAILED' ]
[7/29/2019, 10:06:55 PM] INFO [ 'Trial job OHFjd status changed from WAITING to FAILED' ]
[7/29/2019, 10:06:55 PM] DEBUG [ 'storeTrialJobEvent: event: FAILED, data: undefined, jobDetail: {"id":"OHFjd","status":"FAILED","submitTime":1564459604651,"workingDirectory":"C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","form":{"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7521816105091335, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}","index":0}},"url":"file://localhost:C:\\\\Users\\\\User\\\\nni\\\\experiments\\\\tFYPOJX9\\\\trials\\\\OHFjd","sequenceId":9,"gpuIndices":[],"startTime":1564459611744,"pid":33560,"endTime":1564459615335}' ]
[7/29/2019, 10:06:55 PM] DEBUG [ 'ipcInterface command type: [EN], content:[{"trial_job_id":"OHFjd","event":"FAILED","hyper_params":"{\\"parameter_id\\": 9, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"dropout_rate\\": 0.7521816105091335, \\"conv_size\\": 7, \\"hidden_size\\": 512, \\"batch_size\\": 1, \\"learning_rate\\": 0.0001}, \\"parameter_index\\": 0}"}]' ]
[7/29/2019, 10:06:55 PM] INFO [ 'Change NNIManager status from: NO_MORE_TRIAL to: DONE' ]
[7/29/2019, 10:06:55 PM] DEBUG [ 'ipcInterface command type: [GE], content:[1]' ]
[7/29/2019, 10:06:55 PM] INFO [ 'Experiment done.' ]
[7/29/2019, 10:06:55 PM] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"dropout_rate": 0.6052547440708993, "conv_size": 5, "hidden_size": 512, "batch_size": 8, "learning_rate": 0.01}, "parameter_index": 0}' ]
[7/29/2019, 10:06:58 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:03 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:08 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:13 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:18 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:23 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:28 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:33 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:38 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:43 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:48 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:53 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:07:58 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:03 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:08 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:13 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:18 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:23 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:28 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:33 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:38 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:43 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:48 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:53 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:08:58 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:03 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:08 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:13 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:18 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:23 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:28 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:33 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:38 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:43 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'GET: /version: body:\n{}' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'GET: /trial-jobs: body:\n{}' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'GET: /experiment: body:\n{}' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'getTrialJobsByReplayEvents begin' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'getTrialJobsByReplayEvents done' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 10:09:48 PM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'GET: /check-status: body:\n{}' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'GET: /trial-jobs: body:\n{}' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'GET: /metric-data: body:\n{}' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'getTrialJobsByReplayEvents begin' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'getTrialJobsByReplayEvents done' ]
[7/29/2019, 10:09:49 PM] DEBUG [ 'GET: /experiment: body:\n{}' ]
[7/29/2019, 10:09:53 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:09:58 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:10:03 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:10:08 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:10:13 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:10:18 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]
[7/29/2019, 10:10:23 PM] DEBUG [ 'ipcInterface command type: [PI], content:[]' ]

[anandhperumal]

@ultmaster Let me know if you wanna do a remote connection. I'll make myself available as per your convenience

[ultmaster]

Let's first confirm that there is no python execution in your run.ps1, because on my machine, run.ps1 looks like this. And by the look of your previously shared run.ps1, I can see there is a python there.

And as far as I can tell, the script writes two number (exit code and time) to a state file; but you only provided one (1564039806036) previously.

Remote connection might not be convenient in our company; but if you insist I might be able to do it at home. Contact me by my gmail, which can be found in my Github profile.

[anandhperumal]

I missed the line 9. This is kind of strange when I executed the nnictl command, the state file has only timestamp in it. But executing the run.ps1 the state file has two values : 1 1564461430719 And stderr is empty (executed the nnictl command again after running run.ps1)

[ultmaster]

I think 1 means unsuccessful (0 means success); that is why NNI is showing FAILURE.

Are you sure the line cmd /c python mnist.py 2>C:\Users\User\nni\experiments\JAqxevyD\trials\JLCwt\stderr outputs nothing, like totally nothing? Try to remove the part after 2> to see what happens. Try to make it output, at least, something...

[anandhperumal]

I just tried running it again. This time state file contains only timestamp, there is no LASTEXITCODE in it. And stderr is still empty

Can you tell me what is the default debug port number? And just by triggering nnictl create command and connecting it to debug port, I should be able to debug right? or is there any other server I need to start for debugging?

[ultmaster]

Try this: Delete C:\test if you've already had one. Then replace config_windows.yml with the following content:

authorName: default
experimentName: example_mnist
trialConcurrency: 1
maxExecDuration: 1h
maxTrialNum: 10
#choice: local, remote, pai
trainingServicePlatform: local
searchSpacePath: search_space.json
#choice: true, false
useAnnotation: false
tuner:
  #choice: TPE, Random, Anneal, Evolution, BatchTuner, MetisTuner, GPTuner
  #SMAC (SMAC should be installed through nnictl)
  builtinTunerName: TPE
  classArgs:
    #choice: maximize, minimize
    optimize_mode: maximize
trial:
  command: md C:\test
  codeDir: .
  gpuNum: 0

Check whether or not C:\test has been created.

[anandhperumal]

Do you mean C:\Users\User\nni\test? not C:\test. I deleted C:\Users\User\nni\test and ran the nnictl command. It didn't create the test folder and stderr is as usual empty.

[ultmaster]

My point is C:\test, or D:\test, or any folder. Please make sure that you cannot create any folder with command in config file.

If the test folder is not created, then the subprocess is not launched. That might be worth another digging.

[anandhperumal]

So what do you suggest? the steps I followed, let me know if I missed something:

1. python -m pip install --upgrade nni
2. git clone -b v0.9 https://github.com/Microsoft/nni.git
3. nnictl create --config nni\examples\trials\mnist\config_windows.yml

[ultmaster]

@anandh02 I'm suggesting you to find out whether nni can successfully launch any process, so that I can determine whether it's a Python issue, or a NNI issue. So follow my instructions and try to create C:\test with NNI.

[anandhperumal]

@ultmaster "Please make sure that you cannot create any folder with command in config file." What does that even mean? "I'm suggesting you to find out whether nni can successfully launch any process " how am I suppose check this ? Already tried creating test folder from config file, it didn't create any folder.

[anandhperumal]

I'm trying to install from source code but I'm getting this error, any idea why am I getting this error and how to resolve it?

(venv) G:\RWDI\nni-0.8.1\nni-0.8.1>powershell .\install.ps1
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
Rename-Item : Cannot bind argument to parameter 'Path' because it is null.
At G:\RWDI\nni-0.8.1\nni-0.8.1\install.ps1:100 char:21
+         Rename-Item $unzipYarnDir "nni-yarn"
+                     ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidData: (:) [Rename-Item], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorNullNotAllowed,Microsoft.PowerShell.Commands.RenameItemC
   ommand

Copy-Item : Cannot find path 'G:\RWDI\nni-0.8.1\nni-0.8.1\src\webui\build' because it does not exist.
At G:\RWDI\nni-0.8.1\nni-0.8.1\install.ps1:133 char:1
+ Copy-Item "src\webui\build" $NNI_PKG_FOLDER_STATIC -Recurse
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (G:\RWDI\nni-0.8...src\webui\build:String) [Copy-Item], ItemNotFoundExce
   ption
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.CopyItemCommand

[ultmaster]

According to your description, I think NNI failed to launch run.ps1 since it's not even making any directory.

Why are you using nni-0.8.1? Please keep NNI up-to-date.

Refer to this on how to install from source code on Windows.

Please consider moving to Linux, since NNI on Windows is experimental and the process launching part might still contain some bugs we don't know. Using Linux will save your time of struggling. I was wondering if you can provide us with your system information (such as Windows version, hardware info), so that we might be able to reproduce your issue.

[anandhperumal]

Today I tried installing the nni-0.9 version in my lab machine and I faced the same problem. Later, I migrated to 0.8 and installed through source code and that resolved my issue in the lab machine. But unfortunately, it's not working on my local machine. System config: Windows 10 - 64bit - i7 12gb ram 1050Ti

[ultmaster]

You can try to install by pip install nni==0.8.

We need more specific version details of Windows 10, like 1903 or whatever. Check that in your Settings -> About.

[anandhperumal]

I already tried installing through pip but then I get this error (below) while running the nni

And I can clearly see nni in pip list

Windows 10 version 1809 OS build 17763.557

[ultmaster]

Did you try it under virtualenv? Same thing?

[anandhperumal]

Yes, the problem exists there too.

[ultmaster]

Try to debug with https://github.com/microsoft/nni/blob/b56e4a174ceee5589af9a1249e67febdaa47f0d7/tools/nni_cmd/launcher.py#L59 and determine where nni is installed and why it cannot be found.

I still recommend you switch to Linux since your installation on Windows seems really painstaking for reasons that we are currently not sure of.

[anandhperumal]

Where can I find the debug port number?

[ultmaster]

@anandh02 What do you mean by "debug port number"? We only have port number for webUI. If you want to do a further debugging, set --debug when you execute nnictl. If you want to have a check at our source code, this will be a good start.

[anandhperumal]

When we mention --debug it starts the server in debug mode. So there will be a debug port in which we can connect our python ide and start debugging the source code.

[ultmaster]

@anandh02 I asked my colleagues about this, and according to @liuzhe-lz , there is no such port implemented in NNI currently. You can open another issue elaborating your feature request, and we can discuss about it afterwards.

[anandhperumal]

Sure. Thanks.

[Grzechu11]

Hello all, I have the same problem.

When I run nnictl create --config config.yml or nnictl create --config nni\examples\trials\ mnist\config_windows.yml web nni starts, but all trials have the status failed.

same as Author @anandhperumal stderr file is empty Moreover, I keep getting a pop-up:

-------------------------------------------------- --------------
#! / usr / bin / env bash
# Copyright (C) 2014, Alexey Pavlov
# mailto: alexpux@gmail.com
# This file is part of Minimal SYStem version 2.
# https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$ COMSPEC" "$ @"
-------------------------------------------------- ------------------

Please help me. Thanks

[ultmaster]

@Grzechu11 What's your environment? Windows or Linux? What's your nni version? Can you post your nnimanager.log?

It doesn't look like the same problem to me. Please post a new issue instead of commenting. And please provide more details so that we can help.

[Grzechu11]

Thank you for your answer.

I still think that I have the same problem as described by @anandhperumal (here: https://github.com/microsoft/nni/issues/1367#issue-472712556)

nni Environment:

• nni version: 1.0
• nni mode(local|pai|remote): local
• OS: Windows 10 x64
• python version: 3.7.3
• is conda: yes
• is running in docker?: no

nnimanager.log

[2019-10-9 23:52:26] INFO [ 'Datastore initialization done' ]
[2019-10-9 23:52:26] INFO [ 'Rest server listening on: http://0.0.0.0:8080' ]
[2019-10-9 23:52:26] INFO [ 'RestServer start' ]
[2019-10-9 23:52:26] INFO [ 'Construct local machine training service.' ]
[2019-10-9 23:52:26] INFO [ 'RestServer base port is 8080' ]
[2019-10-9 23:52:27] INFO [ 'NNIManager setClusterMetadata, key: trial_config, value: {"command":"python D:\\\\Projekty\\\\ML_5DWChallenge\\\\vel5\\\\day2\\\\nni_day2.py","codeDir":"D:\\\\Projekty\\\\ML_5DWChallenge\\\\vel5\\\\day2","gpuNum":0}' ]
[2019-10-9 23:52:27] INFO [ 'required GPU number is 0' ]
[2019-10-9 23:52:28] INFO [ 'Starting experiment: f6l0s17p' ]
[2019-10-9 23:52:28] INFO [ 'Change NNIManager status from: INITIALIZED to: RUNNING' ]
[2019-10-9 23:52:28] INFO [ 'Add event listeners' ]
[2019-10-9 23:52:28] INFO [ 'Run local machine training service.' ]
[2019-10-9 23:52:30] INFO [ 'NNIManager received command from dispatcher: ID, ' ]
[2019-10-9 23:52:30] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"k": 5}, "parameter_index": 0}' ]
[2019-10-9 23:52:30] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"k": 6}, "parameter_index": 0}' ]
[2019-10-9 23:52:33] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 0, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"k\\": 5}, \\"parameter_index\\": 0}","index":0}}' ]
[2019-10-9 23:52:33] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 1, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"k\\": 6}, \\"parameter_index\\": 0}","index":0}}' ]
[2019-10-9 23:52:44] INFO [ 'Trial job BqdRb status changed from WAITING to FAILED' ]
[2019-10-9 23:52:44] INFO [ 'Trial job YBCho status changed from WAITING to FAILED' ]
[2019-10-9 23:52:44] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"k": 2}, "parameter_index": 0}' ]
[2019-10-9 23:52:44] INFO [ 'NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"k": 7}, "parameter_index": 0}' ]
[2019-10-9 23:52:49] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 2, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"k\\": 2}, \\"parameter_index\\": 0}","index":0}}' ]
[2019-10-9 23:52:49] INFO [ 'submitTrialJob: form: {"jobType":"TRIAL","hyperParameters":{"value":"{\\"parameter_id\\": 3, \\"parameter_source\\": \\"algorithm\\", \\"parameters\\": {\\"k\\": 7}, \\"parameter_index\\": 0}","index":0}}' ]
[2019-10-9 23:52:54] INFO [ 'Change NNIManager status from: RUNNING to: STOPPING' ]
[2019-10-9 23:52:54] INFO [ 'Stopping experiment, cleaning up ...' ]
[2019-10-9 23:52:56] INFO [ 'Stopping local machine training service...' ]
[2019-10-9 23:52:56] INFO [ 'Change NNIManager status from: STOPPING to: STOPPED' ]
[2019-10-9 23:52:56] INFO [ 'Experiment stopped.' ]

When I run nni, every few seconds a message pops up to open the notebook, which contains information that I do not understand:

-------------------------------------------------- --------------
#! / usr / bin / env bash
# Copyright (C) 2014, Alexey Pavlov
# mailto: alexpux@gmail.com
# This file is part of Minimal SYStem version 2.
# https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$ COMSPEC" "$ @"
-------------------------------------------------- ------------------

In trials i have folder BqdRb and files in it:

• parameter.cfg

  {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"k": 5}, "parameter_index": 0}

• run.ps1 If I want to run the file I have the same as when running nni.

  
  -------------------------------------------------- --------------
  #! / usr / bin / env bash
  # Copyright (C) 2014, Alexey Pavlov
  # mailto: alexpux@gmail.com
  # This file is part of Minimal SYStem version 2.
  # https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
  # File: cmd

"$ COMSPEC" "$ @"

- stderr - is empty

[ultmaster]

-------------------------------------------------- --------------
#! / usr / bin / env bash
# Copyright (C) 2014, Alexey Pavlov
# mailto: alexpux@gmail.com
# This file is part of Minimal SYStem version 2.
# https://sourceforge.net/p/msys2/wiki/MSYS2%20installation/
# File: cmd

"$ COMSPEC" "$ @"

Are you sure this is the content of your run.ps1 file? Not the running output?

[Grzechu11]

Yes, this is output Content is:

cd D:\Projekty\ML_5DWChallenge\vel5\day2
$env:NNI_PLATFORM="local"
$env:NNI_EXP_ID="f6l0s17p"
$env:NNI_SYS_DIR="C:\Users\gdworak\nni\experiments\f6l0s17p\trials\BqdRb"
$env:NNI_TRIAL_JOB_ID="BqdRb"
$env:NNI_OUTPUT_DIR="C:\Users\gdworak\nni\experiments\f6l0s17p\trials\BqdRb"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python D:\Projekty\ML_5DWChallenge\vel5\day2\nni_day2.py >C:\Users\gdworak\nni\experiments\f6l0s17p\trials\BqdRb\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\gdworak\nni\experiments\f6l0s17p\trials\BqdRb\.nni\state -NoNewline -encoding utf8

[ultmaster]

@Grzechu11 On my machine, run.ps1 looks like this:

cd C:\Users\v-yugzh\Downloads\nni\examples\trials\mnist\.
$env:NNI_PLATFORM="local"
$env:NNI_EXP_ID="i1FVjIc4"
$env:NNI_SYS_DIR="C:\Users\v-yugzh\nni\experiments\i1FVjIc4\trials\ulnWM"
$env:NNI_TRIAL_JOB_ID="ulnWM"
$env:NNI_OUTPUT_DIR="C:\Users\v-yugzh\nni\experiments\i1FVjIc4\trials\ulnWM"
$env:NNI_TRIAL_SEQ_ID="0"
$env:MULTI_PHASE="false"
$env:CUDA_VISIBLE_DEVICES="-1"
cmd /c python mnist.py 2>C:\Users\v-yugzh\nni\experiments\i1FVjIc4\trials\ulnWM\stderr
$NOW_DATE = [int64](([datetime]::UtcNow)-(get-date "1/1/1970")).TotalSeconds
$NOW_DATE = "$NOW_DATE" + (Get-Date -Format fff).ToString()
Write $LASTEXITCODE " " $NOW_DATE  | Out-File C:\Users\v-yugzh\nni\experiments\i1FVjIc4\trials\ulnWM\.nni\state -NoNewline -encoding utf8

stderr looks like this:

cmd : Traceback (most recent call last):
At C:\Users\v-yugzh\nni\experiments\i1FVjIc4\trials\ulnWM\run.ps1:10 char:1
+ cmd /c python mnist.py 2>C:\Users\v-yugzh\nni\experiments\i1FVjIc4\tr ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (Traceback (most recent call last)::String) [], RemoteException
    + FullyQualifiedErrorId : NativeCommandError

  File "mnist.py", line 9, in <module>
    import tensorflow as tf
ModuleNotFoundError: No module named 'tensorflow'

Did you try to run run.ps1 directly in your terminal? How does it end up?