search

microsoft / nni

An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
https://nni.readthedocs.io
MIT License
13.99k stars 1.81k forks source link

tuner_params = nni.get_next_parameter() error #4725

Closed leiqing110 closed 2 years ago

leiqing110 commented 2 years ago

Describe the issue: When I don't write the code in the green box, it can't execute normally, and the webui shows that it takes 6 seconds to execute the successed, which is not right. But after I add the green code, the program can execute normally, I feel that there is a problem with the red code, did not get the parameter dictionary image

Environment:

Configuration:

experimentName: NNI_test # An optional name to help you distinguish experiments.

Hyper-parameter search space can either be configured here or in a seperate file.

"config.yml" shows how to specify a seperate search space file.

The common schema of search space is documented here:

https://nni.readthedocs.io/en/stable/Tutorial/SearchSpaceSpec.html

searchSpace: epochs: _type: choice _value: [200,450,600] lr: _type: uniform _value: [0.0016,0.001]

trialCommand: CUDA_VISIBLE_DEVICES=3 python3 train_nni.py ../imagenet/ --model efficientnet_b2 -b 256 --sched step --epochs 200 --decay-epochs 2.4 --decay-rate .97 --opt rmsproptf --opt-eps .001 -j 8 --warmup-lr 1e-6 --weight-decay 1e-5 --drop 0.3 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .016 # The command to launch a trial. NOTE: change "python3" to "python" if you are using Windows. trialCodeDirectory: . # The path of trial code. By default it's ".", which means the same directory of this config file. trialGpuNumber: 1 # How many GPUs should each trial use. CUDA is required when it's greater than zero.

trialConcurrency: 1 # Run 4 trials concurrently. maxTrialNumber: 2 # Generate at most 10 trials. maxExperimentDuration: 20h # Stop generating trials after 1 hour.

tuner: # Configure the tuning algorithm. name: TPE # Supported algorithms: TPE, Random, Anneal, Evolution, GridSearch, GPTuner, PBTTuner, etc.

Full list: https://nni.readthedocs.io/en/latest/Tuner/BuiltinTuner.html

classArgs: # Algorithm specific arguments. See the tuner's doc for details. optimize_mode: maximize # "minimize" or "maximize"

Configure the training platform.

Supported platforms: local, remote, openpai, aml, kubeflow, kubernetes, adl.

trainingService: platform: local useActiveGpu: false # NOTE: Use "true" if you are using an OS with graphical interface (e.g. Windows 10, Ubuntu desktop)

Reason and details: https://nni.readthedocs.io/en/latest/reference/experiment_config.html#useactivegpu

Log message:

How to reproduce it?:

liuzhe-lz commented 2 years ago

Please attach the log files.

leiqing110 commented 2 years ago

Please attach the log files. Thank you so much dispatcher.log [2022-04-01 09:05:19] INFO (nni.experiment/MainThread) Creating experiment, Experiment ID: ^[[36mou6xtbjg^[[0m [2022-04-01 09:05:19] INFO (nni.experiment/MainThread) Starting web server... [2022-04-01 09:05:19] DEBUG (urllib3.connectionpool/MainThread) Starting new HTTP connection (1): localhost:8080 [2022-04-01 09:05:20] DEBUG (urllib3.connectionpool/MainThread) Starting new HTTP connection (1): localhost:8080 [2022-04-01 09:05:20] DEBUG (urllib3.connectionpool/MainThread) http://localhost:8080 "GET /api/v1/nni/check-status HTTP/1.1" 200 36 [2022-04-01 09:05:20] INFO (nni.experiment/MainThread) Setting up... [2022-04-01 09:05:20] DEBUG (urllib3.connectionpool/MainThread) Starting new HTTP connection (1): localhost:8080 [2022-04-01 09:05:20] DEBUG (urllib3.connectionpool/MainThread) http://localhost:8080 "POST /api/v1/nni/experiment HTTP/1.1" 200 28 [2022-04-01 09:05:20] INFO (nni.experiment/MainThread) Web UI URLs: ^[[36mhttp://127.0.0.1:8080 http://172.17.0.3:8080^[[0m [2022-04-01 09:05:20] INFO (nni.tools.nnictl.launcher/MainThread) To stop experiment run "nnictl stop ou6xtbjg" or "nnictl stop --all" [2022-04-01 09:05:20] INFO (nni.tools.nnictl.launcher/MainThread) Reference: https://nni.readthedocs.io/en/stable/Tutorial/Nnictl.html [2022-04-01 09:05:21] DEBUG (nni.main/MainThread) START [2022-04-01 09:05:21] DEBUG (nni.main/MainThread) decoded exp_params: [{"experimentName":"TIMM","searchSpaceFile":"/pytorch-image-models-master/search_space.json","trialCommand":"CUDA_VISIBLE_DEVICES=3 python3 train_nni.py ../imagenet/ --model efficientnet_b2 -b 128 --sched step --epochs 450 --decay-epochs 2.4 --decay-rate .97 --opt rmsproptf --opt-eps .001 -j 8 --warmup-lr 1e-6 --weight-decay 1e-5 --drop 0.3 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .016","trialCodeDirectory":"/pytorch-image-models-master","trialConcurrency":1,"trialGpuNumber":1,"ma@@@

nnictl_stderr.log

Experiment ou6xtbjg start: 2022-04-01 09:05:19.667069

nnimanager.log

[2022-04-01 09:05:19] DEBUG (SqlDB) Database directory: /pytorch-image-models-master/log/ou6xtbjg/db [2022-04-01 09:05:20] INFO (NNIDataStore) Datastore initialization done [2022-04-01 09:05:20] INFO (RestServer) RestServer start [2022-04-01 09:05:20] INFO (RestServer) RestServer base port is 8080 [2022-04-01 09:05:20] INFO (main) Rest server listening on: http://0.0.0.0:8080 [2022-04-01 09:05:20] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:20] DEBUG (NNIRestHandler) POST: /experiment: body: { experimentName: 'TIMM', searchSpaceFile: '/pytorch-image-models-master/search_space.json', searchSpace: { 'epochs:': { _type: 'choice', _value: [Array] }, lr: { _type: 'uniform', _value: [Array] } }, trialCommand: 'CUDA_VISIBLE_DEVICES=3 python3 train_nni.py ../imagenet/ --model efficientnet_b2 -b 128 --sched step --epochs 450 --decay-epochs 2.4 --decay-rate .97 --opt rmsproptf --opt-eps .001 -j 8 --warmup-lr 1e-6 --weight-decay 1e-5 --drop 0.3 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .016', trialCodeDirectory: '/pytorch-image-models-master', trialConcurrency: 1, trialGpuNumber: 1, maxExperimentDuration: '24h', maxTrialNumber: 2, useAnnotation: false, debug: false, logLevel: 'info', experimentWorkingDirectory: '/pytorch-image-models-master/log', tuner: { name: 'TPE', classArgs: { optimize_mode: 'maximize' } }, trainingService: { platform: 'local', trialCommand: 'CUDA_VISIBLE_DEVICES=3 python3 train_nni.py ../imagenet/ --model efficientnet_b2 -b 128 --sched step --epochs 450 --decay-epochs 2.4 --decay-rate .97 --opt rmsproptf --opt-eps .001 -j 8 --warmup-lr 1e-6 --weight-decay 1e-5 --drop 0.3 --model-ema --model-ema-decay 0.9999 --aa rand-m9-mstd0.5 --remode pixel --reprob 0.2 --amp --lr .016', trialCodeDirectory: '/pytorch-image-models-master', trialGpuNumber: 1, debug: false, useActiveGpu: true, maxTrialNumberPerGpu: 1, reuseMode: false } } [2022-04-01 09:05:20] INFO (NNIManager) Starting experiment: ou6xtbjg [2022-04-01 09:05:20] INFO (NNIManager) Setup training service... [2022-04-01 09:05:20] INFO (LocalTrainingService) Construct local machine training service. [2022-04-01 09:05:20] INFO (NNIManager) Setup tuner... [2022-04-01 09:05:20] DEBUG (NNIManager) dispatcher command: python3 -m nni --exp_params eyJleHBlcmltZW50TmFtZSI6IlRJTU0iLCJzZWFyY2hTcGFjZUZpbGUiOiIvcHl0b3JjaC1pbWFnZS1tb2RlbHMtbWFzdGVyL3NlYXJjaF9zcGFjZS5qc29uIiwidHJpYWxDb21tYW5kIjoiQ1VEQV9WSVNJQkxFX0RFVklDRVM9MyBweXRob24zIHRyYWluX25uaS5weSAuLi9pbWFnZW5ldC8gLS1tb2RlbCBlZmZpY2llbnRuZXRfYjIgLWIgMTI4IC0tc2NoZWQgc3RlcCAtLWVwb2NocyA0NTAgLS1kZWNheS1lcG9jaHMgMi40IC0tZGVjYXktcmF0ZSAuOTcgLS1vcHQgcm1zcHJvcHRmIC0tb3B0LWVwcyAuMDAxIC1qIDggLS13YXJtdXAtbHIgMWUtNiAtLXdlaWdodC1kZWNheSAxZS01IC0tZHJvcCAwLjMgIC0tbW9kZWwtZW1hIC0tbW9kZWwtZW1hLWRlY2F5IDAuOTk5OSAtLWFhIHJhbmQtbTktbXN0ZDAuNSAtLXJlbW9kZSBwaXhlbCAtLXJlcHJvYiAwLjIgLS1hbXAgLS1sciAuMDE2IiwidHJpYWxDb2RlRGlyZWN0b3J5IjoiL3B5dG9yY2gtaW1hZ2UtbW9kZWxzLW1hc3RlciIsInRyaWFsQ29uY3VycmVuY3kiOjEsInRyaWFsR3B1TnVtYmVyIjoxLCJtYXhFeHBlcmltZW50RHVyYXRpb24iOiIyNGgiLCJtYXhUcmlhbE51bWJlciI6MiwidXNlQW5ub3RhdGlvbiI6ZmFsc2UsImRlYnVnIjpmYWxzZSwibG9nTGV2ZWwiOiJpbmZvIiwiZXhwZXJpbWVudFdvcmtpbmdEaXJlY3RvcnkiOiIvcHl0b3JjaC1pbWFnZS1tb2RlbHMtbWFzdGVyL2xvZyIsInR1bmVyIjp7Im5hbWUiOiJUUEUiLCJjbGFzc0FyZ3MiOnsib3B0aW1pemVfbW9kZSI6Im1heGltaXplIn19LCJ0cmFpbmluZ1NlcnZpY2UiOnsicGxhdGZvcm0iOiJsb2NhbCIsInRyaWFsQ29tbWFuZCI6IkNVREFfVklTSUJMRV9ERVZJQ0VTPTMgcHl0aG9uMyB0cmFpbl9ubmkucHkgLi4vaW1hZ2VuZXQvIC0tbW9kZWwgZWZmaWNpZW50bmV0X2IyIC1iIDEyOCAtLXNjaGVkIHN0ZXAgLS1lcG9jaHMgNDUwIC0tZGVjYXktZXBvY2hzIDIuNCAtLWRlY2F5LXJhdGUgLjk3IC0tb3B0IHJtc3Byb3B0ZiAtLW9wdC1lcHMgLjAwMSAtaiA4IC0td2FybXVwLWxyIDFlLTYgLS13ZWlnaHQtZGVjYXkgMWUtNSAtLWRyb3AgMC4zICAtLW1vZGVsLWVtYSAtLW1vZGVsLWVtYS1kZWNheSAwLjk5OTkgLS1hYSByYW5kLW05LW1zdGQwLjUgLS1yZW1vZGUgcGl4ZWwgLS1yZXByb2IgMC4yIC0tYW1wIC0tbHIgLjAxNiIsInRyaWFsQ29kZURpcmVjdG9yeSI6Ii9weXRvcmNoLWltYWdlLW1vZGVscy1tYXN0ZXIiLCJ0cmlhbEdwdU51bWJlciI6MSwiZGVidWciOmZhbHNlLCJ1c2VBY3RpdmVHcHUiOnRydWUsIm1heFRyaWFsTnVtYmVyUGVyR3B1IjoxLCJyZXVzZU1vZGUiOmZhbHNlfX0= [2022-04-01 09:05:20] INFO (NNIManager) Change NNIManager status from: INITIALIZED to: RUNNING [2022-04-01 09:05:20] INFO (NNIManager) Add event listeners [2022-04-01 09:05:20] DEBUG (NNIManager) Send tuner command: INITIALIZE: [object Object] [2022-04-01 09:05:20] DEBUG (IpcInterface) ipcInterface command type: [IN], content:[{"epochs:":{"_type":"choice","_value":[10,20]},"lr":{"_type":"uniform","_value":[0.0005,0.0001]}}] [2022-04-01 09:05:20] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:20] INFO (LocalTrainingService) Run local machine training service. [2022-04-01 09:05:20] WARNING (GPUScheduler) gpu_metrics file does not exist! [2022-04-01 09:05:21] INFO (NNIManager) NNIManager received command from dispatcher: ID, [2022-04-01 09:05:21] DEBUG (IpcInterface) ipcInterface command type: [GE], content:[1] [2022-04-01 09:05:21] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00032176811215000077}, "parameter_index": 0} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:24] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:24] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:24] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:24] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /version: body: {} [2022-04-01 09:05:24] DEBUG (NNIRestHandler) GET: /tensorboard-tasks: body: {} [2022-04-01 09:05:25] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:25] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:25] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:25] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:25] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:25] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:25] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:25] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:25] INFO (NNIManager) submitTrialJob: form: { sequenceId: 0, hyperParameters: { value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00032176811215000077}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2022-04-01 09:05:25] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'DkBdw', status: 'WAITING', submitTime: 1648803925873, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/pytorch-image-models-master/log/ou6xtbjg/trials/DkBdw', workingDirectory: '/pytorch-image-models-master/log/ou6xtbjg/trials/DkBdw', form: { sequenceId: 0, hyperParameters: { value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00032176811215000077}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:29] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:29] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /version: body: {} [2022-04-01 09:05:29] DEBUG (NNIRestHandler) GET: /tensorboard-tasks: body: {} [2022-04-01 09:05:30] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:30] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:30] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:30] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:30] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:30] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:30] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:30] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:30] INFO (NNIManager) Trial job DkBdw status changed from WAITING to RUNNING [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:31] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:31] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:31] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:31] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:31] DEBUG (NNIRestHandler) GET: /version: body: {} [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /tensorboard-tasks: body: {} [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:32] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:32] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:32] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:35] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:35] DEBUG (LocalTrainingService) trialJob status update: DkBdw, SUCCEEDED [2022-04-01 09:05:35] INFO (NNIManager) Trial job DkBdw status changed from RUNNING to SUCCEEDED [2022-04-01 09:05:35] DEBUG (IpcInterface) ipcInterface command type: [EN], content:[{"trial_job_id":"DkBdw","event":"SUCCEEDED","hyper_params":"{\"parameter_id\": 0, \"parameter_source\": \"algorithm\", \"parameters\": {\"epochs:\": 20, \"lr\": 0.00032176811215000077}, \"parameter_index\": 0}"}] [2022-04-01 09:05:35] DEBUG (IpcInterface) ipcInterface command type: [GE], content:[1] [2022-04-01 09:05:35] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00022183382282163835}, "parameter_index": 0} [2022-04-01 09:05:40] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:40] INFO (NNIManager) submitTrialJob: form: { sequenceId: 1, hyperParameters: { value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00022183382282163835}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2022-04-01 09:05:40] DEBUG (LocalTrainingService) submitTrialJob: return: LocalTrialJobDetail { id: 'elnBk', status: 'WAITING', submitTime: 1648803940958, startTime: undefined, endTime: undefined, tags: undefined, url: 'file://localhost:/pytorch-image-models-master/log/ou6xtbjg/trials/elnBk', workingDirectory: '/pytorch-image-models-master/log/ou6xtbjg/trials/elnBk', form: { sequenceId: 1, hyperParameters: { value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"epochs:": 20, "lr": 0.00022183382282163835}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } }, pid: undefined, gpuIndices: [] } [2022-04-01 09:05:43] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:43] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:43] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:43] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:43] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:43] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:44] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:45] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:45] INFO (NNIManager) Trial job elnBk status changed from WAITING to RUNNING [2022-04-01 09:05:45] INFO (NNIManager) Change NNIManager status from: RUNNING to: NO_MORE_TRIAL [2022-04-01 09:05:50] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:05:50] DEBUG (LocalTrainingService) trialJob status update: elnBk, SUCCEEDED [2022-04-01 09:05:50] INFO (NNIManager) Trial job elnBk status changed from RUNNING to SUCCEEDED [2022-04-01 09:05:50] DEBUG (IpcInterface) ipcInterface command type: [EN], content:[{"trial_job_id":"elnBk","event":"SUCCEEDED","hyper_params":"{\"parameter_id\": 1, \"parameter_source\": \"algorithm\", \"parameters\": {\"epochs:\": 20, \"lr\": 0.00022183382282163835}, \"parameter_index\": 0}"}] [2022-04-01 09:05:50] INFO (NNIManager) Change NNIManager status from: NO_MORE_TRIAL to: DONE [2022-04-01 09:05:50] INFO (NNIManager) Experiment done. [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:52] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:52] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:52] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:52] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /version: body: {} [2022-04-01 09:05:52] DEBUG (NNIRestHandler) GET: /tensorboard-tasks: body: {} [2022-04-01 09:05:53] DEBUG (NNIRestHandler) GET: /experiment: body: {} [2022-04-01 09:05:53] DEBUG (NNIRestHandler) GET: /experiment-metadata: body: {} [2022-04-01 09:05:53] DEBUG (NNIRestHandler) GET: /trial-jobs: body: {} [2022-04-01 09:05:53] DEBUG (NNIRestHandler) GET: /metric-data: body: {} [2022-04-01 09:05:53] DEBUG (NNIDataStore) getTrialJobsByReplayEvents begin [2022-04-01 09:05:53] DEBUG (NNIDataStore) getTrialJobsByReplayEvents done [2022-04-01 09:05:53] DEBUG (NNIRestHandler) GET: /check-status: body: {} [2022-04-01 09:05:55] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:00] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:05] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:10] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:15] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:20] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:25] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:30] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:35] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:40] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:45] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:50] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:06:55] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:00] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:05] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:10] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:15] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:20] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:25] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:30] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:35] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:40] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:45] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:50] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:07:55] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:00] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:05] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:10] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:15] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:20] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:25] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:30] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:35] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:40] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:45] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:50] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:08:55] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:00] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:05] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:10] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:15] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:20] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:25] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:30] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:35] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[] [2022-04-01 09:09:40] DEBUG (IpcInterface) ipcInterface command type: [PI], content:[]

liuzhe-lz commented 2 years ago

The search space does not contains batch_size. Hopefully you have already found it.