NNI error while enable Assessor CurveFitting

Simba98 commented 2 years ago

Describe the issue:

The Assessor CurveFitting Throw an Error while try to run custom trail.

Environment:

NNI version: 2.7 (nnictl --version)
Training service (local|remote|pai|aml|etc): 4x remote ssh servers
Client OS: Docker 20.10.12 in Ubuntu 20.04.3
Server OS (for remote mode only): Docker 20.10.12 in Ubuntu 20.04.3
Python version: Python 3.7.11 (Remote) Python 3.8.10 (NNI official Docker image)
PyTorch/TensorFlow version: Docker image: pytorch/pytorch:1.10.0-cuda11.3-cudnn8-devel
Is conda/virtualenv/venv used?: Yes, by the pytorch docker
Is running in Docker?: Yes, both client and server

Configuration:

Experiment config (remember to remove secrets!):


trialCommand: python main.py
trialGpuNumber: 1
trialConcurrency: 4
maxTrialNumber: 1000
max_experiment_duration: '14d'
nniManagerIp: IP

tuner: name: MetisTuner classArgs: optimize_mode: maximize

assessor: name: Curvefitting classArgs: epoch_num: 90 start_step: 5 threshold: 0.9 gap: 1

trainingService: platform: remote machineList:

host: 1 ssh_key_file: ~/.ssh/id_rsa useActiveGpu: true maxTrialNumberPerGpu: 1
host: 2 ssh_key_file: ~/.ssh/id_rsa useActiveGpu: true maxTrialNumberPerGpu: 1
host: 3 ssh_key_file: ~/.ssh/id_rsa useActiveGpu: true maxTrialNumberPerGpu: 1

host: 4 ssh_key_file: ~/.ssh/id_rsa useActiveGpu: true maxTrialNumberPerGpu: 1


- Search space:
```yaml
search_space:
log_lr:
_type: uniform
_value: [ -2, 0 ]
momentum:
_type: uniform
_value: [ 0, 1 ]
log_weight_decay:
_type: uniform
_value: [ -5, -3 ]

Log message:

nnimanager.log:

[2022-04-23 16:38:21] INFO (NNIManager) submitTrialJob: form: {
sequenceId: 7,
hyperParameters: {
value: '{"parameter_id":null,"parameter_source":"customized","parameters":{"log_lr":-1,"momentum":0.9,"log_weight_decay":-4}}',
index: 0
}
}
[2022-04-23 16:38:21] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"log_lr": -0.22318303377913984, "momentum": 0.23205394302683402, "log_weight_decay": -4.5493828664737705}, "parameter_index": 0}
[2022-04-23 16:38:21] ERROR (NNIManager) TypeError: Cannot read properties of undefined (reading 'type')
at GpuScheduler.scheduleMachine (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/gpuScheduler.js:31:24)
at TrialDispatcher.trialManagementLoop (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:473:54)
at async Promise.all (index 2)
at async TrialDispatcher.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:246:9)
at async RouterTrainingService.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/routerTrainingService.js:94:16)
at async Promise.all (index 2)
at async NNIManager.run (/usr/local/lib/python3.8/dist-packages/nni_node/core/nnimanager.js:625:9)
[2022-04-23 16:38:21] INFO (NNIManager) Change NNIManager status from: RUNNING to: ERROR

dispatcher.log:

[2022-04-23 16:19:27] INFO (curvefitting_Assessor/Thread-2) List of effective model: []
[2022-04-23 16:19:27] INFO (curvefitting_Assessor/Thread-2) Prediction done. Trial job id = IBCVl, Predict value = None, wait for more information to predict precisely
[2022-04-23 16:19:31] INFO (curvefitting_Assessor/Thread-2) List of effective model: []
[2022-04-23 16:19:31] INFO (curvefitting_Assessor/Thread-2) Prediction done. Trial job id = RzQ8J, Predict value = None, wait for more information to predict precisely
[2022-04-23 16:31:40] INFO (curvefitting_Assessor/Thread-2) List of effective model: []
[2022-04-23 16:31:40] INFO (curvefitting_Assessor/Thread-2) Prediction done. Trial job id = m2Xqa, Predict value = None, wait for more information to predict precisely
[2022-04-23 16:38:11] INFO (curvefitting_Assessor/Thread-2) List of effective model: ['vap', 'pow3', 'linear', 'logx_linear', 'dr_hill_zero_background', 'log_power', 'pow4', 'mmf', 'exp4', 'weibull', 'janoschek']
[2022-04-23 16:38:11] INFO (Metis_Tuner_AutoML/Thread-1) Received trial result.
[2022-04-23 16:38:11] INFO (Metis_Tuner_AutoML/Thread-1) value is : -67.2699966430664
[2022-04-23 16:38:11] INFO (Metis_Tuner_AutoML/Thread-1) parameter is : {'log_lr': -1.1437021747529292, 'momentum': 0.5591691812460651, 'log_weight_decay': -3.353357657094285}
[2022-04-23 16:38:21] INFO (Metis_Tuner_AutoML/Thread-1) Generate paramageters: 
{'log_lr': -0.22318303377913984, 'momentum': 0.23205394302683402, 'log_weight_decay': -4.5493828664737705}
[2022-04-23 16:39:12] INFO (curvefitting_Assessor/Thread-2) Prediction done. Trial job id = ka6mO, Predict value = 68.56728949835005

nnictl stdout and stderr:

> nnictl log stdout b49e1grs
--------------------------------------------------------------------------------
Experiment b49e1grs start: 2022-04-22 08:14:10.874494
--------------------------------------------------------------------------------
> nnictl log stderr b49e1grs
--------------------------------------------------------------------------------
Experiment b49e1grs start: 2022-04-22 08:14:10.874494
--------------------------------------------------------------------------------
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:56: RuntimeWarning: overflow encountered in power
return c - a * x**(-alpha)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:147: RuntimeWarning: overflow encountered in power
return a/(1.+(x/np.exp(b))**c)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:171: RuntimeWarning: overflow encountered in power
return c - (a*x+b)**-alpha
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:171: RuntimeWarning: invalid value encountered in power
return c - (a*x+b)**-alpha
/usr/local/lib/python3.8/dist-packages/scipy/optimize/_minpack_py.py:833: OptimizeWarning: Covariance of the parameters could not be estimated
warnings.warn('Covariance of the parameters could not be estimated',
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:220: RuntimeWarning: overflow encountered in exp
return c - np.exp(-a*(x**alpha)+b)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:242: RuntimeWarning: divide by zero encountered in true_divide
return c - a / np.log(x)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:242: RuntimeWarning: divide by zero encountered in double_scalars
return c - a / np.log(x)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:124: RuntimeWarning: invalid value encountered in true_divide
return (theta * x**eta) / (kappa**eta + x**eta)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:196: RuntimeWarning: invalid value encountered in power
return alpha - (alpha - beta) / (1. + (kappa * x)**delta)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:56: RuntimeWarning: overflow encountered in multiply
return c - a * x**(-alpha)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:124: RuntimeWarning: invalid value encountered in double_scalars
return (theta * x**eta) / (kappa**eta + x**eta)
TypeError: Cannot read properties of undefined (reading 'type')
at GpuScheduler.scheduleMachine (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/gpuScheduler.js:31:24)
at TrialDispatcher.trialManagementLoop (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:473:54)
at async Promise.all (index 2)
at async TrialDispatcher.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:246:9)
at async RouterTrainingService.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/routerTrainingService.js:94:16)
at async Promise.all (index 2)
at async NNIManager.run (/usr/local/lib/python3.8/dist-packages/nni_node/core/nnimanager.js:625:9) {
cause: TypeError: Cannot read properties of undefined (reading 'type')
  at GpuScheduler.scheduleMachine (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/gpuScheduler.js:31:24)
  at TrialDispatcher.trialManagementLoop (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:473:54)
  at async Promise.all (index 2)
  at async TrialDispatcher.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/trialDispatcher.js:246:9)
  at async RouterTrainingService.run (/usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/routerTrainingService.js:94:16)
  at async Promise.all (index 2)
  at async NNIManager.run (/usr/local/lib/python3.8/dist-packages/nni_node/core/nnimanager.js:625:9)
}
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:124: RuntimeWarning: overflow encountered in double_scalars
return (theta * x**eta) / (kappa**eta + x**eta)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:124: RuntimeWarning: overflow encountered in power
return (theta * x**eta) / (kappa**eta + x**eta)
/usr/local/lib/python3.8/dist-packages/nni/algorithms/hpo/curvefitting_assessor/curvefunctions.py:124: RuntimeWarning: overflow encountered in multiply
return (theta * x**eta) / (kappa**eta + x**eta)

How to reproduce it?:

Run the experiment
Start a custom trail when all environments are running
Waiting for one remote environment to complete
Expected run custom train, actually run the hyperparameters by runner and error.

liuzhe-lz commented 2 years ago

Curve fitting only accepts accuracy metrics. The result value cannot be negative.

Simba98 commented 2 years ago

Curve fitting only accepts accuracy metrics. The result value cannot be negative.

The metrics are accuracies. However, perhaps Metis Tuner makes it negative? As you can see,

tuner:
name: MetisTuner
classArgs:
optimize_mode: maximize

How to migrate these existing trainings to a new experiment? Some tuners don't accept manual trails. Are manual trails meaningful for the metis tuner?

liuzhe-lz commented 2 years ago

Oh, sorry, the log's value is inversed by tuner. Though curve fitting has reported a lot of warnings, the real error is raised by GPU scheduler. I'll look into that.

liuzhe-lz commented 2 years ago

I'm currently unsure if we will make a patch release. You can fix it locally by editting /usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/gpuScheduler.js line 31. Add a line: constraint = constraint ?? {type: 'None', gpus: []}

Simba98 commented 2 years ago

I'm currently unsure if we will make a patch release. You can fix it locally by editting /usr/local/lib/python3.8/dist-packages/nni_node/training_service/reusable/gpuScheduler.js line 31. Add a line: constraint = constraint ?? {type: 'None', gpus: []}

Thank you for your reply. I will try it if this fixes the issue.

microsoft / nni

NNI error while enable Assessor CurveFitting #4797