Open LiangqunLu opened 4 years ago
INFO:tensorflow:Oracle triggered exit
C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\_methods.py:151: RuntimeWarning: invalid value encountered in reduce
ret = umr_sum(arr, axis, dtype, out, keepdims)
C:\ProgramData\Anaconda3\lib\site-packages\numpy\core\_methods.py:183: RuntimeWarning: invalid value encountered in reduce
arrmean = umr_sum(arr, axis, dtype, keepdims=True)
don't know whether I am facing the same issue
Facing the same issue. Get 'INFO:tensorflow:Oracle triggered exit' just after all trials complete. My code:
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, random_state=0) # 116883 rows in X, y ak = StructuredDataRegressor(max_trials=1, seed=0) ak.fit(X_train, y_train, epochs=100) ak.evaluate(X_test, y_test) ak.export_model().save('ak')
In addition, I get the following when trying to save the exported model:
ak.export_model().save('ak') WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.iter WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_1 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.beta_2 WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.decay WARNING:tensorflow:Unresolved object in checkpoint: (root).optimizer.learning_rate WARNING:tensorflow:A checkpoint was restored (e.g. tf.train.Checkpoint.restore or tf.keras.Model.load_weights) but not all checkpointed values were used. See above for specific issues. Use expect_partial() on the load status object, e.g. tf.train.Checkpoint.restore(...).expect_partial(), to silence these warnings, or use assert_consumed() to make the check explicit. See https://www.tensorflow.org/guide/checkpoint#loading_mechanics for details. 2020-06-16 23:58:37.896026: W tensorflow/python/util/util.cc:319] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1786: calling BaseResourceVariable.init (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. Traceback (most recent call last): File "
", line 1, in File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/save.py", line 115, in save_model signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/saving/saved_model/save.py", line 78, in save save_lib.save(model, filepath, signatures, options) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 909, in save meta_graph_def, saveable_view, signatures, options.namespace_whitelist) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 553, in _fill_meta_graph_def object_map, resource_map, asset_info = saveable_view.map_resources() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/saved_model/save.py", line 251, in map_resources new_resource = obj._create_resource() File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/lookup_ops.py", line 1932, in _create_resource name=self._name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/gen_lookup_ops.py", line 1113, in mutable_dense_hash_table_v2 max_load_factor=max_load_factor, name=name) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/op_def_library.py", line 468, in _apply_op_helper preferred_dtype=default_dtype) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/framework/ops.py", line 1280, in convert_to_tensor raise RuntimeError("Attempting to capture an EagerTensor without " RuntimeError: Attempting to capture an EagerTensor without building a function.
I can't view the summary of the exported model either:
model = ak.export_model() model.summary() Model: "model"
Layer (type) Output Shape Param #
input_1 (InputLayer) [(None, 6)] 0
Traceback (most recent call last): File "
", line 1, in File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/network.py", line 1310, in summary print_fn=print_fn) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/layer_utils.py", line 226, in print_summary print_layer_summary(layers[i]) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/utils/layer_utils.py", line 184, in print_layer_summary fields = [name + ' (' + cls_name + ')', output_shape, layer.count_params()] File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1607, in count_params return int(sum(np.prod(w.shape.as_list()) for w in self.weights)) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/engine/base_layer.py", line 1607, in return int(sum(np.prod(w.shape.as_list()) for w in self.weights)) AttributeError: 'TrackableWeightHandler' object has no attribute 'shape'
Issue solved in the latest version.
@utkarshgupta137 @haifeng-jin This issue is absolutely not solved in the latest version. I'm still getting this on 1.0.8. Can you please re-open this issue?
@jisho-iemoto okay, no problem
Here's a minimal reproduction for the issue. code:
import autokeras as ak
import numpy as np
x = input_node = ak.Input()
x = ak.RNNBlock()(x)
x = ak.TemporalReduction()(x)
output_node = ak.RegressionHead(loss='cosine_similarity')(x)
reg = ak.AutoModel(
inputs=input_node,
outputs=output_node,
overwrite=True,
max_trials=1)
reg.fit(x=np.zeros((100, 50, 20)), y=np.ones((100, 20)), epochs=2)
output:
Trial 1 Complete [00h 00m 10s]
val_loss: -0.86031574010849
Best val_loss So Far: -0.86031574010849
Total elapsed time: 00h 00m 10s
INFO:tensorflow:Oracle triggered exit
Epoch 1/2
4/4 [==============================] - 0s 19ms/step - loss: -0.5827 - mean_squared_error: 0.9994
Epoch 2/2
4/4 [==============================] - 0s 18ms/step - loss: -0.8673 - mean_squared_error: 0.9979
Is it an error that exits the program completely? or it is keep running to the end and you can still export the found best model?
when I run
it always triggers "INFO:tensorflow:Oracle triggered exit", then the worst model is selected for the evaluation.
Apparently, I expected the autokeras to select the optimized parameters and have the best model for evaluation.