search

keras-team / autokeras

AutoML library for deep learning
http://autokeras.com/
Apache License 2.0
9.16k stars 1.4k forks source link

Bug: WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint. Not sure how to solve #1831

Closed arturdaraujo closed 1 year ago

arturdaraujo commented 1 year ago

Bug Description

I only encounter this problem when I use validation data. I get the following code Warning:

Bug Reproduction

path = "/mnt/d/python_directory_2/autokeras_models/"
search = ak.StructuredDataRegressor(max_trials=1, overwrite=True, directory=path)
search.fit(
    x=train_data.drop("lod_price", axis=1),
    y=train_data["lod_price"],
    verbose=2,
    validation_data=(
        val_data.drop("lod_price", axis=1),
        val_data["lod_price"],
    ),
    workers=8,
    use_multiprocessing=True,
)
[...]
235/235 - 1s - loss: 1.1315e-04 - mean_squared_error: 1.1315e-04 - val_loss: 0.0057 - val_mean_squared_error: 0.0057 - 850ms/epoch - 4ms/step
Epoch 40/1000
235/235 - 1s - loss: 1.0554e-04 - mean_squared_error: 1.0554e-04 - val_loss: 0.0054 - val_mean_squared_error: 0.0054 - 905ms/epoch - 4ms/step
Trial 1 Complete [00h 00m 53s]
val_loss: 0.00468598073348403

Best val_loss So Far: 0.00468598073348403
Total elapsed time: 00h 00m 53s
INFO:tensorflow:Oracle triggered exit
INFO:tensorflow:Assets written to: /mnt/d/python_directory_2/autokeras_models/structured_data_regressor/best_model/assets
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_1
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_2
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate

When I try to run my prediction I get this error:

y_pred = search.predict(test_data.drop("lod_price", axis=1))

2023-01-12 00:44:55.827191: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2023-01-12 00:44:55.827271: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
[...]
2023-01-12 00:44:55.828525: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
2023-01-12 00:44:55.828560: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at lookup_table_op.cc:929 : FAILED_PRECONDITION: Table not initialized.
Traceback (most recent call last):

  File "/tmp/ipykernel_2758/1795775326.py", line 1, in <cell line: 1>
    y_pred = search.predict(test_data.drop("lod_price", axis=1))

  File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/autokeras/tasks/structured_data.py", line 165, in predict
    return super().predict(x=x, **kwargs)

  File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/autokeras/auto_model.py", line 455, in predict
    y = model.predict(dataset, **kwargs)

  File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None

  File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,

FailedPreconditionError: Graph execution error:

Detected at node 'model/multi_category_encoding/string_lookup_101/None_Lookup/LookupTableFindV2' defined at (most recent call last):
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/runpy.py", line 196, in _run_module_as_main
      return _run_code(code, main_globals, None,
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/runpy.py", line 86, in _run_code
[...]
packages/tensorflow/python/framework/op_def_library.py", line 797, in _apply_op_helper
      op = g._create_op_internal(op_type_name, inputs, dtypes=None,
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/tensorflow/python/framework/func_graph.py", line 694, in _create_op_internal
      return super(FuncGraph, self)._create_op_internal(  # pylint: disable=protected-access
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3754, in _create_op_internal
      ret = Operation(
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 2133, in __init__
      self._traceback = tf_stack.extract_stack_for_node(self._c_op)
    File "/home/artur/miniforge3/envs/py310_autokeras/lib/python3.10/site-packages/tensorflow/python/util/tf_stack.py", line 183, in extract_stack_for_node
      return _tf_stack.extract_stack_for_node(
Node: 'model/multi_category_encoding/string_lookup_101/None_Lookup/LookupTableFindV2'
Table not initialized.
     [[{{node model/multi_category_encoding/string_lookup_101/None_Lookup/LookupTableFindV2}}]] [Op:__inference_predict_function_868340]

Data used by the code:

Expected Behavior

Setup Details

Include the details about the versions of:

Additional context