An open source AutoML toolkit for automate machine learning lifecycle, including feature engineering, neural architecture search, model compression and hyper-parameter tuning.
Describe the issue:
metrics are exist in the .nni/metrics file, but are not shown in the web UI.
Environment:
NNI version: 2.10.1
Training service (local|remote|pai|aml|etc): local
Client OS: ubuntu
Server OS (for remote mode only):
Python version: 3.8
PyTorch/TensorFlow version: pytorch 1.11.0
Is conda/virtualenv/venv used?: conda
Is running in Docker?:
Configuration:
Experiment config (remember to remove secrets!):
Search space:
Log message:
nnimanager.log:
[2023-09-08 21:48:22] INFO (main) Start NNI manager
[2023-09-08 21:48:22] INFO (NNIDataStore) Datastore initialization done
[2023-09-08 21:48:22] INFO (RestServer) Starting REST server at port 8080, URL prefix: "/"
[2023-09-08 21:48:22] WARNING (NNITensorboardManager) Tensorboard may not installed, if you want to use tensorboard, please check if tensorboard installed.
[2023-09-08 21:48:22] INFO (RestServer) REST server started.
[2023-09-08 21:48:23] INFO (NNIManager) Starting experiment: ex5bd2uk
[2023-09-08 21:48:23] INFO (NNIManager) Setup training service...
[2023-09-08 21:48:23] INFO (LocalTrainingService) Construct local machine training service.
[2023-09-08 21:48:23] INFO (NNIManager) Setup tuner...
[2023-09-08 21:48:23] INFO (NNIManager) Change NNIManager status from: INITIALIZED to: RUNNING
[2023-09-08 21:48:23] INFO (NNIManager) Add event listeners
[2023-09-08 21:48:23] INFO (LocalTrainingService) Run local machine training service.
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: ID,
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.0001}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 21, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 13, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 16, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 18, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 21:48:23] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 16, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 0,
hyperParameters: {
value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.0001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 1,
hyperParameters: {
value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 21, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 2,
hyperParameters: {
value: '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 13, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 3,
hyperParameters: {
value: '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 4,
hyperParameters: {
value: '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 16, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 5,
hyperParameters: {
value: '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 6,
hyperParameters: {
value: '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 18, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:28] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 7,
hyperParameters: {
value: '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 16, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 21:48:38] INFO (NNIManager) Trial job qMWBG status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job zcCbb status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job HCtVc status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job ABjjO status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job SP0Vs status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job sD2KG status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job HSWik status changed from WAITING to RUNNING
[2023-09-08 21:48:38] INFO (NNIManager) Trial job Wtx7j status changed from WAITING to RUNNING
[2023-09-08 22:15:26] INFO (NNIManager) Trial job HCtVc status changed from RUNNING to SUCCEEDED
[2023-09-08 22:15:26] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 18, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:15:31] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 8,
hyperParameters: {
value: '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 18, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:15:36] INFO (NNIManager) Trial job cLUCA status changed from WAITING to RUNNING
[2023-09-08 22:17:19] INFO (NNIManager) Trial job Wtx7j status changed from RUNNING to SUCCEEDED
[2023-09-08 22:17:19] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 11, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:17:24] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 9,
hyperParameters: {
value: '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 11, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:17:30] INFO (NNIManager) Trial job NvOlF status changed from WAITING to RUNNING
[2023-09-08 22:17:45] INFO (NNIManager) Trial job SP0Vs status changed from RUNNING to SUCCEEDED
[2023-09-08 22:17:45] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 19, "learning_rate": 0.0001}, "parameter_index": 0}
[2023-09-08 22:17:50] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 10,
hyperParameters: {
value: '{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 19, "learning_rate": 0.0001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:17:55] INFO (NNIManager) Trial job p9B3c status changed from WAITING to RUNNING
[2023-09-08 22:18:00] INFO (NNIManager) Trial job HSWik status changed from RUNNING to SUCCEEDED
[2023-09-08 22:18:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 11, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 22:18:05] INFO (NNIManager) Trial job sD2KG status changed from RUNNING to SUCCEEDED
[2023-09-08 22:18:05] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 11,
hyperParameters: {
value: '{"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 11, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:18:05] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:18:10] INFO (NNIManager) Trial job TUgDG status changed from WAITING to RUNNING
[2023-09-08 22:18:10] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 12,
hyperParameters: {
value: '{"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:18:15] INFO (NNIManager) Trial job GHMqA status changed from WAITING to RUNNING
[2023-09-08 22:18:30] INFO (NNIManager) Trial job zcCbb status changed from RUNNING to SUCCEEDED
[2023-09-08 22:18:30] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 12, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:18:35] INFO (NNIManager) Trial job ABjjO status changed from RUNNING to SUCCEEDED
[2023-09-08 22:18:35] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 13,
hyperParameters: {
value: '{"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 12, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:18:35] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 14, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 22:18:41] INFO (NNIManager) Trial job IYhpN status changed from WAITING to RUNNING
[2023-09-08 22:18:41] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 14,
hyperParameters: {
value: '{"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 14, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:18:46] INFO (NNIManager) Trial job EbPy0 status changed from WAITING to RUNNING
[2023-09-08 22:26:30] INFO (NNIManager) Trial job qMWBG status changed from RUNNING to SUCCEEDED
[2023-09-08 22:26:30] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 15, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:26:35] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 15,
hyperParameters: {
value: '{"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 20, "k": 15, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:26:40] INFO (NNIManager) Trial job cxlmg status changed from WAITING to RUNNING
[2023-09-08 22:43:43] INFO (NNIManager) Trial job cLUCA status changed from RUNNING to SUCCEEDED
[2023-09-08 22:43:44] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 15, "learning_rate": 0.0001}, "parameter_index": 0}
[2023-09-08 22:43:49] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 16,
hyperParameters: {
value: '{"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 15, "learning_rate": 0.0001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:43:54] INFO (NNIManager) Trial job LPQlu status changed from WAITING to RUNNING
[2023-09-08 22:44:25] INFO (NNIManager) Trial job NvOlF status changed from RUNNING to SUCCEEDED
[2023-09-08 22:44:25] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:44:30] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 17,
hyperParameters: {
value: '{"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 17, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:44:35] INFO (NNIManager) Trial job KeQ1g status changed from WAITING to RUNNING
[2023-09-08 22:45:32] INFO (NNIManager) Trial job TUgDG status changed from RUNNING to SUCCEEDED
[2023-09-08 22:45:32] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 11, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:45:37] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 18,
hyperParameters: {
value: '{"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 11, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:45:42] INFO (NNIManager) Trial job HdAvr status changed from WAITING to RUNNING
[2023-09-08 22:46:18] INFO (NNIManager) Trial job IYhpN status changed from RUNNING to SUCCEEDED
[2023-09-08 22:46:18] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}
[2023-09-08 22:46:23] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 19,
hyperParameters: {
value: '{"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 30, "k": 19, "learning_rate": 0.0005}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:46:28] INFO (NNIManager) Trial job N1jtt status changed from WAITING to RUNNING
[2023-09-08 22:46:49] INFO (NNIManager) Trial job EbPy0 status changed from RUNNING to SUCCEEDED
[2023-09-08 22:46:49] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 21, "learning_rate": 0.001}, "parameter_index": 0}
[2023-09-08 22:46:54] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 20,
hyperParameters: {
value: '{"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"fs_update_frequency": 15, "k": 21, "learning_rate": 0.001}, "parameter_index": 0}',
index: 0
},
placementConstraint: { type: 'None', gpus: [] }
}
[2023-09-08 22:46:59] INFO (NNIManager) Trial job jZPGa status changed from WAITING to RUNNING
dispatcher.log:
[2023-09-08 21:48:23] INFO (nni.tuner.tpe/MainThread) Using random seed 1760022900
[2023-09-08 21:48:23] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started
[2023-09-08 22:18:00] INFO (nni.common.hpo_utils.dedup/Thread-1) Tuning algorithm generated duplicate parameter: {('fs_update_frequency',): 0, ('k',): 5, ('learning_rate',): 1}
[2023-09-08 22:18:00] INFO (nni.common.hpo_utils.dedup/Thread-1) Use grid search for deduplication.
[2023-09-08 22:18:00] INFO (nni.tuner.gridsearch/Thread-1) Grid initialized, size: (3×12×3) = 108
Describe the issue: metrics are exist in the .nni/metrics file, but are not shown in the web UI.
Environment:
Configuration:
Log message:
How to reproduce it?: