Model saving doesn't work for StructuredDataRegressor

[jayavanth]

Bug Description

I can't save the best model as h5 or even a tf file

Bug Reproduction

Code for reproducing the bug:

model = regressor.export_model()

model.save('testmodel.h5')

gives me

NotImplementedError: Save or restore weights that is not an instance of `tf.Variable` is not supported in h5, use `save_format='tf'` instead. Got a model or layer CategoricalEncoding with weights ....

It suggests I use tf instead of h5, but when I do that I get this error

model.save('testmodel', save_format='tf')

2020-04-03 14:44:31.041453: W tensorflow/python/util/util.cc:329] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.
*** RuntimeError: Attempting to capture an EagerTensor without building a function.

Data used by the code:

Expected Behavior

Save the best model in either h5 or tf format

Setup Details

• OS type and version: Ubuntu 18.04.01
• Python: 3.6.6
• autokeras: 1.0.2
• scikit-learn: 0.20.3
• numpy: 1.18.2
• pandas: 1.0.1
• tensorflow: 2.2.0-dev20200401 (nightly)

Additional context

I also can't load the model json after saving it but I think that's related to #1023

[Kraks]

I play with the regression example from the autokeras website and have the same issue when saving to tf format, but a different one to h5 format:

Traceback (most recent call last):
  File "auto_ex.py", line 47, in <module>
    model.save("m.h5")
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/engine/network.py", line 1008, in save
    signatures, options)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 112, in save_model
    model, filepath, overwrite, include_optimizer)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 109, in save_model_to_hdf5
    save_weights_to_hdf5_group(model_weights_group, model_layers)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 627, in save_weights_to_hdf5_group
    weight_values = K.batch_get_value(weights)
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3270, in batch_get_value
    return [x.numpy() for x in tensors]
  File "/home/kraks/.local/lib/python3.6/site-packages/tensorflow_core/python/keras/backend.py", line 3270, in <listcomp>
    return [x.numpy() for x in tensors]
AttributeError: 'TrackableWeightHandler' object has no attribute 'numpy'

Versions: OS: Linux Mint 19.3 kernel 5.0.0-32-generic Python 3.6.9 tensorflow: 2.1.0 pandas: 1.0.3 autokeras: 1.0.2 scikit-learn: 0.22.2.post1 bumpy: 1.18.2

[jayavanth]

@Kraks Try this and see if it works. It works for me and I can save it but I can't load it back.

https://stackoverflow.com/questions/52357542/attributeerror-tensor-object-has-no-attribute-numpy

[cordeirojoao]

Hello, I'm having the same issue :( Does anyone was able to export a model create with StructuredDataClassifier api??

model3.save("modelo3", save_format='tf')

Traceback (most recent call last)

in 1 print("tipo do modelo gravado", type(model3)) 2 #tf.keras.models.save_model(model1, "modelo1") ## ----> 3 model3.save("modelo3", save_format='tf') ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\engine\network.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options) 1006 """ 1007 save.save_model(self, filepath, overwrite, include_optimizer, save_format, -> 1008 signatures, options) 1009 1010 def save_weights(self, filepath, overwrite=True, save_format=None): ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\saving\save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options) 113 else: 114 saved_model_save.save(model, filepath, overwrite, include_optimizer, --> 115 signatures, options) 116 117 ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\keras\saving\saved_model\save.py in save(model, filepath, overwrite, include_optimizer, signatures, options) 76 # we use the default replica context here. 77 with distribution_strategy_context._get_default_replica_context(): # pylint: disable=protected-access ---> 78 save_lib.save(model, filepath, signatures, options) 79 80 if not include_optimizer: ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in save(obj, export_dir, signatures, options) 907 object_saver = util.TrackableSaver(checkpoint_graph_view) 908 asset_info, exported_graph = _fill_meta_graph_def( --> 909 meta_graph_def, saveable_view, signatures, options.namespace_whitelist) 910 saved_model.saved_model_schema_version = ( 911 constants.SAVED_MODEL_SCHEMA_VERSION) ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in _fill_meta_graph_def(meta_graph_def, saveable_view, signature_functions, namespace_whitelist) 551 resource_initializer_ops = [] 552 with exported_graph.as_default(): --> 553 object_map, resource_map, asset_info = saveable_view.map_resources() 554 for resource_initializer_function in resource_initializer_functions: 555 asset_dependencies = [] ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\saved_model\save.py in map_resources(self) 249 # pylint: disable=protected-access 250 with ops.device(obj._resource_device): --> 251 new_resource = obj._create_resource() 252 # pylint: enable=protected-access 253 resource_map[obj.resource_handle] = new_resource ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\ops\lookup_ops.py in _create_resource(self) 1930 value_shape=self._value_shape, 1931 initial_num_buckets=self._initial_num_buckets, -> 1932 name=self._name) 1933 if context.executing_eagerly(): 1934 self._table_name = None ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\ops\gen_lookup_ops.py in mutable_dense_hash_table_v2(empty_key, deleted_key, value_dtype, container, shared_name, use_node_name_sharing, value_shape, initial_num_buckets, max_load_factor, name) 1111 value_shape=value_shape, 1112 initial_num_buckets=initial_num_buckets, -> 1113 max_load_factor=max_load_factor, name=name) 1114 _result = _outputs[:] 1115 if _execute.must_record_gradient(): ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\framework\op_def_library.py in _apply_op_helper(op_type_name, name, **keywords) 466 dtype=dtype, 467 as_ref=input_arg.is_ref, --> 468 preferred_dtype=default_dtype) 469 except TypeError as err: 470 if dtype is None: ~\AppData\Local\conda\conda\envs\Env2\lib\site-packages\tensorflow_core\python\framework\ops.py in convert_to_tensor(value, dtype, name, as_ref, preferred_dtype, dtype_hint, ctx, accepted_result_types) 1278 graph = get_default_graph() 1279 if not graph.building_function: -> 1280 raise RuntimeError("Attempting to capture an EagerTensor without " 1281 "building a function.") 1282 return graph.capture(value, name=name) RuntimeError: Attempting to capture an EagerTensor without building a function.

[cordeirojoao]

I tried to use import tensorflow as tf tf.enable_eager_execution()

But autokeras is running of TF 2.0 that doesn't work that way, I think. @jayavanth Can you please let me know how you did it?

@Kraks Try this and see if it works. It works for me and I can save it but I can't load it back.

https://stackoverflow.com/questions/52357542/attributeerror-tensor-object-has-no-attribute-numpy

[pacohernandezg]

Same problem here, when I try to save the exported model from the StructuredDataClassifier example, I get the following error:

RuntimeError: Attempting to capture an EagerTensor without building a function.

Thank you.

[KartikChugh]

I can save but loading gives me ValueError: Unknown layer: CategoricalEncoding

[ogabrielluiz]

I'm having the same problem.

[mattou78400]

I'm also having the issue with StructuredDataInput

[alexfdo]

I want to offer a solution that I use myself It is based on Pickle + using the directories in the class structure at https://autokeras.com/structured_data_regressor/

 Class StructuredDataRegressor
 autokeras.StructuredDataRegressor (
   ...............................
      metrics = no
      project_name = "structd_data_regressor",
      max_trials = 100,
      directory = no,
     **kwargs
 )

If you do not specify your names for directories, then the system uses the default values, each time overwriting the information in the same directory ./structd_data_regressor
I use this variant to set my own names for directories.

model = ak.StructuredDataRegressor(max_trials=max_trials,
                                                ....................................................
                                               project_name=model_name,
                                               directory='data/models_saved_data/'
                                               )

model_name - this is the name under which the model will be saved. And this same model_name I indicate to pickle which model to load. I save the model completely, without any operation such as export_model.

[utkarshgupta137]

@alexfdo I'm facing the same issue. Could you elaborate on what you've done? I tried to provide a directory myself, but still getting the same error.

[alexfdo]

@utkarshgupta137

save and load model

def load_pickle_model_from_file(name_file):
    with open(name_file, 'rb') as file:
        loaded_model = pickle.load(file)
    return loaded_model

def save_pickle_model_to_file(model, name_model):
    with open(Path(str(name_model)+".pickle"), 'wb') as file:
        pickle.dump(model, file)
    print(str(Path(str(name_model)+".pickle"))+' - model saved!')

create: model_name - you select own name; directory - you select place for model's data

model = ak.StructuredDataRegressor(max_trials=max_trials,
                                               column_names=data_cols,
                                               column_types=data_type,
                                               project_name=model_name
                                               directory='data/models_saved_data/'
                                               )

save after creating: model_name - your selected name on previouse stage

save_pickle_model_to_file(model, model_name)

[belkov0912]

I have the same error. I'm adding DenseHashTable in DNNModel, intend to save embeddings in it, this is code:

`class DNNModel(tf.keras.Model):
def init(self, xxx): self.table = self._create_embedding_table(vocabulary_list, embeddings)

def _create_embedding_table(self, vocab_list, embeddings):
    dimension = embeddings.shape[1]
    table = tf.lookup.experimental.DenseHashTable(tf.string, tf.float32, [2.0]*dimension, "empty_key", "deleted_key")
    # 对比，测试到底注释前后有没有效果
    keys = tf.constant([i for i in vocab_list], tf.string)
    values = tf.convert_to_tensor(embeddings, tf.float64)
    values = tf.cast(values, tf.float32)
    table.insert(keys, values)
    return table

def call(self, inputs, training=None, mask=None):
    fc_embeddings = self._input_layer(inputs)
    bert_embeddings = self._look_up(inputs)
    net = tf.concat([fc_embeddings, bert_embeddings], axis=1)
   ....

` all is ok when runn it, but when i export model with the code tf.keras.models.save_model(model, FLAGS.servable_model_dir), it will raise the same error:RuntimeError: Attempting to capture an EagerTensor without building a function.

I debug the code, it likes serialize the DenseHashTable in dnnModel will raise Exceptions:

File "/Users/jiananliu/work/neirongrecom/model/ctr_model/wide_n_deep/wide_n_deep_keras_main.py", line 1021, in run tf.keras.models.save_model(model, FLAGS.servable_model_dir, include_optimizer=False) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/keras/saving/save.py", line 138, in save_model signatures, options) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/keras/saving/saved_model/save.py", line 78, in save save_lib.save(model, filepath, signatures, options) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 951, in save obj, export_dir, signatures, options, meta_graph_def) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 1027, in _build_meta_graph options.namespace_whitelist) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 595, in _fill_meta_graph_def object_map, resource_map, asset_info = saveable_view.map_resources() File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/saved_model/save.py", line 270, in map_resources new_resource = new_obj._create_resource() File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/ops/lookup_ops.py", line 1945, in _create_resource name=self._name) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/ops/gen_lookup_ops.py", line 1113, in mutable_dense_hash_table_v2 max_load_factor=max_load_factor, name=name) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 470, in _apply_op_helper preferred_dtype=default_dtype) File "/Users/jiananliu/anaconda3/envs/transformer/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 1307, in convert_to_tensor raise RuntimeError("Attempting to capture an EagerTensor without " RuntimeError: Attempting to capture an EagerTensor without building a function.

[haifeng-jin]

As I tested the latest version of AutoKeras can save the structureddataregressor fine.

[utkarshgupta137]

Issue solved in the latest version, thanks.

[Samar-080301]

Issue solved in the latest version, thanks.

In autokeras ==1.0.8?

[gbmax]

Saving model with StructruedDataRegressor example(California house ex.) produces same error..

I tried to save tf-format, keras-format, convert to sequential model and save, all results the same.

seq_model = tf.keras.Sequential(model.layers)
seq_model.save("model_test/h5_california.h5", save_format="h5")

haifeng-jin said saving model is OK, so I guess it is because of packages version difference. Below is my test spec., different? or another package dependency?

< test spec. > autokeras.version = 1.0.8 tensorflow.version = 2.3.0 python = 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)

< source - almost from autokeras example code except model.save >

from sklearn.datasets import fetch_california_housing
import numpy as np
import pandas as pd
import tensorflow as tf
import autokeras as ak

house_dataset = fetch_california_housing()
df = pd.DataFrame(
    np.concatenate((
        house_dataset.data, 
        house_dataset.target.reshape(-1,1)),
        axis=1),
    columns=house_dataset.feature_names + ['Price'])
train_size = int(df.shape[0] * 0.9)
df[:train_size].to_csv('train.csv', index=False)
df[train_size:].to_csv('eval.csv', index=False)
train_file_path = 'train.csv'
test_file_path = 'eval.csv'

import pandas as pd
import numpy as np
x_train = pd.read_csv('train.csv')
y_train = x_train.pop('Price')

x_test = pd.read_csv('eval.csv')
y_test = x_test.pop('Price')

reg = ak.StructuredDataRegressor(max_trials=3, overwrite=True)
reg.fit(x_train, y_train, epochs=10)

model = reg.export_model()
model.save("model_test/h5_california.h5", save_format="h5")

< Error Message >

---

NotImplementedError Traceback (most recent call last)

in ----> 1 seq_model.save("model_test/h5_california.h5", save_format="h5") 2 # tf.keras.models.save_model(model, "model_test/pb_california", save_format="tf") ~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options) 1977 """ 1978 save.save_model(self, filepath, overwrite, include_optimizer, save_format, -> 1979 signatures, options) 1980 1981 def save_weights(self, ~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options) 129 'or using `save_weights`.') 130 hdf5_format.save_model_to_hdf5( --> 131 model, filepath, overwrite, include_optimizer) 132 else: 133 saved_model_save.save(model, filepath, overwrite, include_optimizer, ~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in save_model_to_hdf5(model, filepath, overwrite, include_optimizer) 117 model_weights_group = f.create_group('model_weights') 118 model_layers = model.layers --> 119 save_weights_to_hdf5_group(model_weights_group, model_layers) 120 121 # TODO(b/128683857): Add integration tests between tf.keras and external ~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in save_weights_to_hdf5_group(f, layers) 633 for layer in sorted(layers, key=lambda x: x.name): 634 g = f.create_group(layer.name) --> 635 weights = _legacy_weights(layer) 636 weight_values = K.batch_get_value(weights) 637 weight_names = [w.name.encode('utf8') for w in weights] ~/miniconda3/envs/ak/lib/python3.6/site-packages/tensorflow/python/keras/saving/hdf5_format.py in _legacy_weights(layer) 882 'Save or restore weights that is not an instance of `tf.Variable` is ' 883 'not supported in h5, use `save_format=\'tf\'` instead. Got a model ' --> 884 'or layer {} with weights {}'.format(layer.__class__.__name__, weights)) 885 return weights NotImplementedError: Save or restore weights that is not an instance of `tf.Variable` is not supported in h5, use `save_format='tf'` instead. Got a model or layer MultiCategoryEncoding with weights [, , ]

[haifeng-jin]

Does the tf format work? As we tested, it should work. The h5 may not at the moment.

[gbmax]

Thanks for good work. Yes, tf format works. let me ask another question. I am using autokeras for generating prediction model efficiently(StructuredData). For portable device deployment, I have to use TensorflowLite or Tensorflowjs. I tested various combination of format and convert tool. Unfortunately, even keras-generated model from keras homepage example fails to convert. (https://keras.io/examples/structured_data/structured_data_classification_from_scratch/) conversion script:

import tensorflowjs as tfjs
tfjs.converters.save_keras_model(model, 'model_test/tfjs_model')

BTW what I want to know clear is, autokeras-generated model would be just keras model, so that if keras model is fine for converion to tflite or tfjs format, autokeras model would be OK also. Am I right?

update : comment posted below was deleted because the issue is different, sorry for that : ).

[haifeng-jin]

I am not very familiar with tfjs or tf lite. I am not sure why it doesn't convert.

For the exported Keras model from AutoKeras, it is just a Keras Model. It should convert if other Keras Models do.