can't load models from a tuner that finished search

[GonyRosenman]

was doing HP optimization with keras tuner, then my notebook erroneously got deleted.

now trying to reverse the process and i have the folder with the oracle and all the trials in my drive.

the tuner itself is recognized succesfully.

when running :

tuner.get_best_models(num_models=20)

i got an error. something that i assume got to do with my new build_model function being not exactly identical to how i originally wrote it (had to extract it from my dis-functioning memory).

i tried to change more things (slight changes to the architecture of the network. to find the original one that won't cause problems) and i suspect i made some irreversible change to the oracle because it didn't load the tuner, instead started a serach and then ran into an error.

now when try to load the tuner i get the following output:

INFO:tensorflow:Reloading Oracle from existing project /content/drive/MyDrive/curiosity gony/intro_to_kt2/oracle.json
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0.state_variables
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0._index_lookup_layer
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-1.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-2._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-4._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer-5._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5._inbound_nodes
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-5.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0._index_lookup_layer.state_variables
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-0._index_lookup_layer._table
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer.cell
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer.state_spec
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer.cell
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer.state_spec
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.forward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).layer_with_weights-3.backward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-1.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-5.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-5.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.forward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.forward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.forward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.backward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.backward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'm' for (root).layer_with_weights-3.backward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-1.embeddings
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-5.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-5.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.forward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.forward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.forward_layer.cell.bias
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.backward_layer.cell.kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.backward_layer.cell.recurrent_kernel
WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer's state 'v' for (root).layer_with_weights-3.backward_layer.cell.bias
WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details.
INFO:tensorflow:Reloading Tuner from /content/drive/MyDrive/curiosity gony/intro_to_kt2/tuner0.json

and when trying to run search:

Search: Running Trial #1

Hyperparameter    |Value             |Best Value So Far 
max_tokens        |6000              |14000             
word_embedding_dim|16                |64                
word_embedding_...|32                |32                
drop              |0.5               |0.1               
tuner/epochs      |4                 |12                
tuner/initial_e...|0                 |0                 
tuner/bracket     |1                 |0                 
tuner/round       |0                 |0                 

Epoch 1/4
---------------------------------------------------------------------------
UnimplementedError                        Traceback (most recent call last)
<ipython-input-22-6681487cc101> in <module>()
----> 1 tuner.search(trainset, epochs = 10, validation_data = valset, callbacks = [Early_stop,reduce_LR])
      2 
      3 models = tuner.get_best_models(num_models=20)
      4 best_hps = tuner.get_best_hyperparameters(num_trials = 20)
      5 model1 = models[0]

12 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     58     ctx.ensure_initialized()
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:
     62     if name is not None:

UnimplementedError: 2 root error(s) found.
  (0) Unimplemented:  Cast string to float is not supported
     [[node sequential/Cast (defined at /usr/local/lib/python3.6/dist-packages/kerastuner/engine/tuner.py:141) ]]
  (1) Cancelled:  Function was cancelled before it was started
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_92177]

do you recognize this situation? is it possible to reverse? most importantly, how do you load a model without the original build_model function?

[GonyRosenman]

i managed to go back to the previous status of simply not being able to load the models. i think that's due to the fact my new build_model function is slightly different than how i wrote it in the first place.

the error i'm getting is the following:

WARNING:tensorflow:Inconsistent references when loading the checkpoint into this object graph. Either the Trackable object references in the Python program have changed in an incompatible way, or the checkpoint was generated in an incompatible program.

Two checkpoint references resolved to different objects (<tensorflow.python.keras.layers.core.Dense object at 0x7fdaa08d7a58> and <tensorflow.python.keras.layers.core.Dropout object at 0x7fdaa08d78d0>).
WARNING:tensorflow:Inconsistent references when loading the checkpoint into this object graph. Either the Trackable object references in the Python program have changed in an incompatible way, or the checkpoint was generated in an incompatible program.

Two checkpoint references resolved to different objects (<tensorflow.python.keras.layers.core.Dense object at 0x7fdaa08d7978> and <tensorflow.python.keras.layers.core.Dense object at 0x7fdaa08d7a58>).