All Trials Failing in NNI Hyperparameter Optimization for TensorFlow Keras Model

Describe the issue: I am currently facing an issue with NNI hyperparameter optimization, where all trials are failing for my deep learning model implemented in TensorFlow Keras. I have attempted to troubleshoot the issue, but have not been able to resolve it.

Environment:

NNI version: 2.10
Training service (local|remote|pai|aml|etc): local
OS: Linux
Python version: 3.9.10
TensorFlow version: 2.12.0

Configuration:

my_config.yaml

authorName: default
experimentName: HPO_opp                 # An optional name to distinguish the experiments
trialConcurrency: 30  #30                         # Run 30 trials concurrently
maxExecDuration: 72h #24h                   # Stop generating all trials after 24 hour
maxTrialNum: 2000 #1000                         # Generate at most 1000 trials
trainingServicePlatform: local
searchSpacePath: search_space.yaml           # Specify the Search Space file path
useAnnotation: false
tuner:                                       # Configure the tuning algorithm
builtinTunerName: TPE
classArgs:                                 # Algorithm specific arguments
optimize_mode: maximize                  # maximize or minimize the needed metrics
trial:
command: python3.9 main.py
codeDir: .
gpuNum: 0

Search space:

search_space.yaml

batch_size:
  _type: choice
  _value: [32, 64, 128, 256]
lr:
_type: choice
_value: [0.001, 0.0001, 0.00001, 0.000001]
number_filters:
_type: choice
_value: [ 8, 16, 32, 64, 128]
lstm_units:
  _type: choice
  _value: [8, 16, 32, 64, 128]
se_ratio:
  _type: choice
  _value: [2, 4, 8, 12, 16]
epochs:
  _type: choice
  _value: [100, 150, 200]
dropout_re:
  _type: uniform
  _value: [0.1, 0.9]
dropout_rl:
  _type: uniform
  _value: [0.1, 0.9]

Log message:

nnimanager.log:
`[2023-06-24 17:53:58] INFO (main) Start NNI manager [2023-06-24 17:53:58] INFO (NNIDataStore) Datastore initialization done [2023-06-24 17:53:58] INFO (RestServer) Starting REST server at port 8080, URL prefix: "/" [2023-06-24 17:53:58] INFO (RestServer) REST server started. [2023-06-24 17:53:59] INFO (NNIManager) Starting experiment: 76auh20t [2023-06-24 17:53:59] INFO (NNIManager) Setup training service... [2023-06-24 17:53:59] INFO (LocalTrainingService) Construct local machine training service. [2023-06-24 17:53:59] INFO (NNIManager) Setup tuner... [2023-06-24 17:53:59] INFO (NNIManager) Change NNIManager status from: INITIALIZED to: RUNNING [2023-06-24 17:54:00] INFO (NNIManager) Add event listeners [2023-06-24 17:54:00] INFO (LocalTrainingService) Run local machine training service. [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: ID, [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 64, "se_ratio": 12, "epochs": 150, "dropout_re": 0.16591864960751013, "dropout_rl": 0.4763146846586611}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.6057169343580696, "dropout_rl": 0.5111807849178145}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 16, "lstm_units": 8, "se_ratio": 12, "epochs": 150, "dropout_re": 0.6613944077437538, "dropout_rl": 0.5725532639189658}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 128, "lstm_units": 32, "se_ratio": 8, "epochs": 150, "dropout_re": 0.8053835606971804, "dropout_rl": 0.2622962167703542}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 0.0001, "number_filters": 32, "lstm_units": 32, "se_ratio": 8, "epochs": 150, "dropout_re": 0.7277585495412221, "dropout_rl": 0.6564937586382231}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 128, "lstm_units": 16, "se_ratio": 12, "epochs": 200, "dropout_re": 0.39517840139899085, "dropout_rl": 0.26854125144151225}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 128, "se_ratio": 12, "epochs": 100, "dropout_re": 0.1841927917941133, "dropout_rl": 0.47352908354760426}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 32, "lstm_units": 64, "se_ratio": 2, "epochs": 100, "dropout_re": 0.3055066728555471, "dropout_rl": 0.31631737085163136}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 32, "lstm_units": 16, "se_ratio": 12, "epochs": 100, "dropout_re": 0.6481290589561779, "dropout_rl": 0.12747682346800993}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 16, "lstm_units": 16, "se_ratio": 12, "epochs": 150, "dropout_re": 0.157123756171152, "dropout_rl": 0.6661388926384979}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 8, "lstm_units": 16, "se_ratio": 8, "epochs": 150, "dropout_re": 0.3769323355204677, "dropout_rl": 0.5548874261897466}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 8, "lstm_units": 64, "se_ratio": 4, "epochs": 100, "dropout_re": 0.4478227737102637, "dropout_rl": 0.259258541899661}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 32, "lstm_units": 128, "se_ratio": 16, "epochs": 150, "dropout_re": 0.3741027602289725, "dropout_rl": 0.738503482954283}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 0.0001, "number_filters": 8, "lstm_units": 8, "se_ratio": 16, "epochs": 150, "dropout_re": 0.18279657976959243, "dropout_rl": 0.2211171254575091}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 0.0001, "number_filters": 16, "lstm_units": 64, "se_ratio": 16, "epochs": 200, "dropout_re": 0.83193112053378, "dropout_rl": 0.3378530446379638}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 64, "lstm_units": 8, "se_ratio": 16, "epochs": 200, "dropout_re": 0.45229936995142805, "dropout_rl": 0.16864490993862508}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-06, "number_filters": 8, "lstm_units": 64, "se_ratio": 8, "epochs": 100, "dropout_re": 0.4403294669165829, "dropout_rl": 0.8048131574081352}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 0.001, "number_filters": 64, "lstm_units": 64, "se_ratio": 8, "epochs": 100, "dropout_re": 0.14053290507723848, "dropout_rl": 0.8154379398901324}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 32, "lstm_units": 128, "se_ratio": 12, "epochs": 100, "dropout_re": 0.27090706785320934, "dropout_rl": 0.8011419699392456}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.0001, "number_filters": 8, "lstm_units": 64, "se_ratio": 8, "epochs": 200, "dropout_re": 0.785147629696647, "dropout_rl": 0.3929221184930054}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.5572265916959482, "dropout_rl": 0.4669669945961905}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.5655914186270796, "dropout_rl": 0.4195039249388205}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 64, "se_ratio": 2, "epochs": 150, "dropout_re": 0.5641184558278448, "dropout_rl": 0.5421049357002191}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 32, "se_ratio": 4, "epochs": 150, "dropout_re": 0.8952401834461272, "dropout_rl": 0.6069972137151097}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.6642118356893263, "dropout_rl": 0.6960813527651869}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 128, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.26514027548776387, "dropout_rl": 0.3828924977176382}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 2, "epochs": 150, "dropout_re": 0.6145161790677852, "dropout_rl": 0.4926857904482082}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 32, "se_ratio": 12, "epochs": 150, "dropout_re": 0.7150771003702765, "dropout_rl": 0.8993566416245227}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 16, "lstm_units": 8, "se_ratio": 4, "epochs": 200, "dropout_re": 0.10111042258587734, "dropout_rl": 0.5910534139609338}, "parameter_index": 0} [2023-06-24 17:54:00] INFO (NNIManager) NNIManager received command from dispatcher: TR, {"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-05, "number_filters": 64, "lstm_units": 8, "se_ratio": 12, "epochs": 150, "dropout_re": 0.5258082558809223, "dropout_rl": 0.43259246527006917}, "parameter_index": 0} [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 0, hyperParameters: { value: '{"parameter_id": 0, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 64, "se_ratio": 12, "epochs": 150, "dropout_re": 0.16591864960751013, "dropout_rl": 0.4763146846586611}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 1, hyperParameters: { value: '{"parameter_id": 1, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.6057169343580696, "dropout_rl": 0.5111807849178145}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 2, hyperParameters: { value: '{"parameter_id": 2, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 16, "lstm_units": 8, "se_ratio": 12, "epochs": 150, "dropout_re": 0.6613944077437538, "dropout_rl": 0.5725532639189658}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 3, hyperParameters: { value: '{"parameter_id": 3, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 128, "lstm_units": 32, "se_ratio": 8, "epochs": 150, "dropout_re": 0.8053835606971804, "dropout_rl": 0.2622962167703542}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 4, hyperParameters: { value: '{"parameter_id": 4, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 0.0001, "number_filters": 32, "lstm_units": 32, "se_ratio": 8, "epochs": 150, "dropout_re": 0.7277585495412221, "dropout_rl": 0.6564937586382231}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 5, hyperParameters: { value: '{"parameter_id": 5, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 128, "lstm_units": 16, "se_ratio": 12, "epochs": 200, "dropout_re": 0.39517840139899085, "dropout_rl": 0.26854125144151225}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 6, hyperParameters: { value: '{"parameter_id": 6, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 128, "se_ratio": 12, "epochs": 100, "dropout_re": 0.1841927917941133, "dropout_rl": 0.47352908354760426}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 7, hyperParameters: { value: '{"parameter_id": 7, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 32, "lstm_units": 64, "se_ratio": 2, "epochs": 100, "dropout_re": 0.3055066728555471, "dropout_rl": 0.31631737085163136}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 8, hyperParameters: { value: '{"parameter_id": 8, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 32, "lstm_units": 16, "se_ratio": 12, "epochs": 100, "dropout_re": 0.6481290589561779, "dropout_rl": 0.12747682346800993}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 9, hyperParameters: { value: '{"parameter_id": 9, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 16, "lstm_units": 16, "se_ratio": 12, "epochs": 150, "dropout_re": 0.157123756171152, "dropout_rl": 0.6661388926384979}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 10, hyperParameters: { value: '{"parameter_id": 10, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 8, "lstm_units": 16, "se_ratio": 8, "epochs": 150, "dropout_re": 0.3769323355204677, "dropout_rl": 0.5548874261897466}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 11, hyperParameters: { value: '{"parameter_id": 11, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 8, "lstm_units": 64, "se_ratio": 4, "epochs": 100, "dropout_re": 0.4478227737102637, "dropout_rl": 0.259258541899661}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 12, hyperParameters: { value: '{"parameter_id": 12, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.001, "number_filters": 32, "lstm_units": 128, "se_ratio": 16, "epochs": 150, "dropout_re": 0.3741027602289725, "dropout_rl": 0.738503482954283}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 13, hyperParameters: { value: '{"parameter_id": 13, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 0.0001, "number_filters": 8, "lstm_units": 8, "se_ratio": 16, "epochs": 150, "dropout_re": 0.18279657976959243, "dropout_rl": 0.2211171254575091}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 14, hyperParameters: { value: '{"parameter_id": 14, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 0.0001, "number_filters": 16, "lstm_units": 64, "se_ratio": 16, "epochs": 200, "dropout_re": 0.83193112053378, "dropout_rl": 0.3378530446379638}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 15, hyperParameters: { value: '{"parameter_id": 15, "parameter_source": "algorithm", "parameters": {"batch_size": 128, "lr": 1e-06, "number_filters": 64, "lstm_units": 8, "se_ratio": 16, "epochs": 200, "dropout_re": 0.45229936995142805, "dropout_rl": 0.16864490993862508}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 16, hyperParameters: { value: '{"parameter_id": 16, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-06, "number_filters": 8, "lstm_units": 64, "se_ratio": 8, "epochs": 100, "dropout_re": 0.4403294669165829, "dropout_rl": 0.8048131574081352}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 17, hyperParameters: { value: '{"parameter_id": 17, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 0.001, "number_filters": 64, "lstm_units": 64, "se_ratio": 8, "epochs": 100, "dropout_re": 0.14053290507723848, "dropout_rl": 0.8154379398901324}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 18, hyperParameters: { value: '{"parameter_id": 18, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 1e-06, "number_filters": 32, "lstm_units": 128, "se_ratio": 12, "epochs": 100, "dropout_re": 0.27090706785320934, "dropout_rl": 0.8011419699392456}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 19, hyperParameters: { value: '{"parameter_id": 19, "parameter_source": "algorithm", "parameters": {"batch_size": 64, "lr": 0.0001, "number_filters": 8, "lstm_units": 64, "se_ratio": 8, "epochs": 200, "dropout_re": 0.785147629696647, "dropout_rl": 0.3929221184930054}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 20, hyperParameters: { value: '{"parameter_id": 20, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.5572265916959482, "dropout_rl": 0.4669669945961905}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 21, hyperParameters: { value: '{"parameter_id": 21, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.5655914186270796, "dropout_rl": 0.4195039249388205}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 22, hyperParameters: { value: '{"parameter_id": 22, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 64, "se_ratio": 2, "epochs": 150, "dropout_re": 0.5641184558278448, "dropout_rl": 0.5421049357002191}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 23, hyperParameters: { value: '{"parameter_id": 23, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 32, "se_ratio": 4, "epochs": 150, "dropout_re": 0.8952401834461272, "dropout_rl": 0.6069972137151097}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 24, hyperParameters: { value: '{"parameter_id": 24, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.6642118356893263, "dropout_rl": 0.6960813527651869}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 25, hyperParameters: { value: '{"parameter_id": 25, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-05, "number_filters": 128, "lstm_units": 64, "se_ratio": 4, "epochs": 150, "dropout_re": 0.26514027548776387, "dropout_rl": 0.3828924977176382}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 26, hyperParameters: { value: '{"parameter_id": 26, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 64, "lstm_units": 64, "se_ratio": 2, "epochs": 150, "dropout_re": 0.6145161790677852, "dropout_rl": 0.4926857904482082}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 27, hyperParameters: { value: '{"parameter_id": 27, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 128, "lstm_units": 32, "se_ratio": 12, "epochs": 150, "dropout_re": 0.7150771003702765, "dropout_rl": 0.8993566416245227}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 28, hyperParameters: { value: '{"parameter_id": 28, "parameter_source": "algorithm", "parameters": {"batch_size": 32, "lr": 1e-06, "number_filters": 16, "lstm_units": 8, "se_ratio": 4, "epochs": 200, "dropout_re": 0.10111042258587734, "dropout_rl": 0.5910534139609338}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:05] INFO (NNIManager) submitTrialJob: form: { sequenceId: 29, hyperParameters: { value: '{"parameter_id": 29, "parameter_source": "algorithm", "parameters": {"batch_size": 256, "lr": 1e-05, "number_filters": 64, "lstm_units": 8, "se_ratio": 12, "epochs": 150, "dropout_re": 0.5258082558809223, "dropout_rl": 0.43259246527006917}, "parameter_index": 0}', index: 0 }, placementConstraint: { type: 'None', gpus: [] } } [2023-06-24 17:54:10] INFO (NNIManager) Trial job mMrzB status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job lnLt8 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job bgKvv status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job K3IEF status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job fCN64 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job ooTck status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job KjJtT status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job CDoGB status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job KiAjp status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job okSOD status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job i6mmz status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job ZyArQ status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job IyITY status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job a3lP7 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job EgY81 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job WCSbL status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job XKFpl status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job IwrY1 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job aETRy status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job pl4Rd status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job FphkX status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job IkwBf status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job Z0F01 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job IAmRY status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job WVQ75 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job dE8M2 status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job ESOkL status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job L6XMM status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job xAQ9o status changed from WAITING to FAILED [2023-06-24 17:54:10] INFO (NNIManager) Trial job X6jvr status changed from WAITING to FAILED
dispatcher.log: [2023-06-24 17:54:00] INFO (nni.tuner.tpe/MainThread) Using random seed 1506687620 [2023-06-24 17:54:00] INFO (nni.runtime.msg_dispatcher_base/MainThread) Dispatcher started

nnictl stdout and stderr:

---------------------------------------------------------------------------------
Experiment 76auh20t start: 2023-06-24 17:53:58.406176
--------------------------------------------------------------------------------

microsoft / nni

All Trials Failing in NNI Hyperparameter Optimization for TensorFlow Keras Model #5617