can't predict after saving then loading model

[nttgithubntt]

Hj, Thanks for creating tfrs. I examine a recommendation model based on tfrs, after that, I fit, predict and save model ok, but when loading model with tf.keras.models.load_model, the summary after loading model:

when predicting: the loaded model show error:

ValueError: Could not find matching function to call loaded from the SavedModel. Got: Positional arguments (3 total):

• {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float32>}
• False

• None Keyword arguments: {}

  Expected these arguments to match one of the following 4 option(s).....

How to solve this issues? Tks again.

[maciejkula]

This most commonly happens where the features you pass to call in training are different than the features you pass to predict. For example, you might accidentally pass labels or example weights to call during training, but these are not available in prediction.

To avoid this, make sure you only pass the features you have in prediction into call when defining your compute_loss method.

Can you provide all the options that the saved model error gives you?

[nttgithubntt]

Hj @maciejkula , thanks you for your's responding. Here are the ranking model:

class RankingModel(tfrs.models.Model):

def __init__(self, user_ids, topic_ids, card_ids, card_titles):
    # We take the loss weights in the constructor: this allows us to instantiate
    # several model objects with different loss weights.
    super().__init__()

    self.query_model = UserModel(user_ids, topic_ids)
    self.candidate_model = CardModel(card_ids, card_titles, topic_ids)

    # A small model to take in user and card embedding and predict interactions.
    # We can make this as complicated as we want as long as we output a vector
    # as our prediction. A vector prediction is [Favorites, Shares, Discussions_Started, Press]
    self.interacting_model = tf.keras.Sequential([
        tf.keras.layers.Dense(64, activation="relu"),
        tf.keras.layers.Dense(32, activation="relu"),
        tf.keras.layers.Dense(4),
    ])

    # The ranking tasks.
    self.ranking_task: tf.keras.layers.Layer = tfrs.tasks.Ranking(
        loss=tf.keras.losses.MeanSquaredError(),
        metrics=[tf.keras.metrics.RootMeanSquaredError()]
    )

def call(self, inputs):
    print("call-ranking model inputs info:")
    print(inputs)
    user_embeddings = self.query_model(inputs)
    card_embeddings = self.candidate_model(inputs)
    return self.interacting_model(tf.concat([user_embeddings, card_embeddings], axis=1))

def compute_loss(self, features: Dict[Text, tf.Tensor], training=False) -> tf.Tensor:
    inputs = features.copy()
    del inputs["favorites"], inputs["share"], inputs["discussions_started"], inputs["press"]
    interacting_predictions = self(inputs)
    interacting = tf.concat(
        [tf.expand_dims(features["favorites"], -1),
         tf.expand_dims(features["share"], -1),
         tf.expand_dims(features["discussions_started"], -1),
         tf.expand_dims(features["press"], -1)
         ], axis=1)

    return self.ranking_task(
        labels=interacting,
        predictions=interacting_predictions
    )

model = RankingModel(user_ids=vocabularies[0], topic_ids=vocabularies[1], card_ids=vocabularies[2], card_titles=vocabularies[3]) model.compile(optimizer=tf.keras.optimizers.Adagrad(0.1))

train, test = self.data_manager.get_train_test_batch_set() model.fit(train.batch(8192), epochs=3) print("ranking model summary:") print(self._model.summary())

print("save ranking model:") model.save(DEEP_MODEL_PATH)

model = tf.keras.models.load_model(DEEP_MODEL_PATH)

Check its architecture

print("loaded ranking model ok") print("loaded ranking model summary:") print(model.summary())

predict_dataset = DeepDataManager.get_df_score(user_info, df_card).batch(5) results = model.predict(predict_dataset)

Here are the logs:

/home/ai/Documents/AI/gh-ai/eurekai/bin/python /home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py 2020-10-06 11:58:08.502124: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 /home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py:23: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False. self.data_manager = DeepDataManager(data_path=data_path) 2020-10-06 11:58:10.596744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1 2020-10-06 11:58:10.662514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: pciBusID: 0000:17:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2020-10-06 11:58:10.663019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 1 with properties: pciBusID: 0000:65:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5 coreClock: 1.77GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s 2020-10-06 11:58:10.663046: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1 2020-10-06 11:58:10.663244: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcublas.so.10'; dlerror: libcublas.so.10: cannot open shared object file: No such file or directory 2020-10-06 11:58:10.663287: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory 2020-10-06 11:58:10.663326: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcurand.so.10'; dlerror: libcurand.so.10: cannot open shared object file: No such file or directory 2020-10-06 11:58:10.663365: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory 2020-10-06 11:58:10.663403: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcusparse.so.10'; dlerror: libcusparse.so.10: cannot open shared object file: No such file or directory 2020-10-06 11:58:10.666135: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7 2020-10-06 11:58:10.666155: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1753] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform. Skipping registering GPU devices... 2020-10-06 11:58:10.666427: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags. 2020-10-06 11:58:10.678109: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 3299990000 Hz 2020-10-06 11:58:10.679010: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5cb4090 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2020-10-06 11:58:10.679039: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2020-10-06 11:58:10.680719: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1257] Device interconnect StreamExecutor with strength 1 edge matrix: 2020-10-06 11:58:10.680732: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1263]
fitting ranking model: Epoch 1/3 WARNING:tensorflow:Layer ranking_model is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2. The layer has dtype float64 because its dtype defaults to floatx.

If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float32 by default, call tf.keras.backend.set_floatx('float32'). To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

call-ranking model inputs info: {'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>} WARNING:tensorflow:Layer ranking is casting an input tensor from dtype float32 to the layer's dtype of float64, which is new behavior in TensorFlow 2. The layer has dtype float64 because its dtype defaults to floatx.

If you intended to run this layer in float64, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float32 by default, call tf.keras.backend.set_floatx('float32'). To change just this layer, pass dtype='float32' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.

call-ranking model inputs info: {'user_id': <tf.Tensor 'IteratorGetNext:9' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'IteratorGetNext:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'IteratorGetNext:10' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'IteratorGetNext:6' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'IteratorGetNext:2' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'IteratorGetNext:3' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'Cast:0' shape=(None,) dtype=float64>} 1/1 [==============================] - 0s 389us/step - root_mean_squared_error: 12.4154 - loss: 154.1427 - regularization_loss: 0.0000e+00 - total_loss: 154.1427 Epoch 2/3 1/1 [==============================] - 0s 435us/step - root_mean_squared_error: 12.1651 - loss: 147.9898 - regularization_loss: 0.0000e+00 - total_loss: 147.9898 Epoch 3/3 1/1 [==============================] - 0s 442us/step - root_mean_squared_error: 11.2242 - loss: 125.9836 - regularization_loss: 0.0000e+00 - total_loss: 125.9836 ranking model summary: Model: "ranking_model"

---

Layer (type) Output Shape Param #

user_model (UserModel) multiple 5248

---

card_model (CardModel) multiple 957640

---

sequential_5 (Sequential) (None, 4) 4644

---

ranking (Ranking) multiple 2

Total params: 967,534 Trainable params: 967,532 Non-trainable params: 2

---

None save ranking model: call-ranking model inputs info: {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>} call-ranking model inputs info: {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>} call-ranking model inputs info: {'user_id': <tf.Tensor 'user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'card_recency:0' shape=(None,) dtype=float64>} call-ranking model inputs info: {'user_id': <tf.Tensor 'inputs_5:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None,) dtype=float64>} WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. WARNING:tensorflow:From /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:2309: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. call-ranking model inputs info: {'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>} call-ranking model inputs info: {'user_id': <tf.Tensor 'inputs/user_id:0' shape=(None,) dtype=string>, 'card_id': <tf.Tensor 'inputs/card_id:0' shape=(None,) dtype=string>, 'view_time': <tf.Tensor 'inputs/view_time:0' shape=(None,) dtype=int32>, 'interested_topic_id': <tf.Tensor 'inputs/interested_topic_id:0' shape=(None,) dtype=string>, 'card_title': <tf.Tensor 'inputs/card_title:0' shape=(None,) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs/card_topic_id:0' shape=(None,) dtype=string>, 'card_recency': <tf.Tensor 'inputs/card_recency:0' shape=(None,) dtype=float64>} 2020-10-06 11:58:13.487382: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. WARNING:tensorflow:5 out of the last 5 calls to <function recreate_function..restored_function_body at 0x7f4ab8ff99d8> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has experimental_relax_shapes=True option that relaxes argument shapes that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/tutorials/customization/performance#python_or_tensor_args and https://www.tensorflow.org/api_docs/python/tf/function for more details. loaded ranking model ok loaded ranking model summary: Model: "ranking_model"

---

Layer (type) Output Shape Param #

user_model (UserModel) multiple 5248

---

card_model (CardModel) multiple 957640

---

sequential_5 (Sequential) (None, 4) 4644

---

ranking (Ranking) multiple 2

Total params: 967,534 Trainable params: 967,532 Non-trainable params: 2

---

None sys:1: DtypeWarning: Columns (7,16,17,18,19,20,21,23) have mixed types.Specify dtype option on import or set low_memory=False. /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:69: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df_interactions["user_id"] = user_info["user_id"] /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:70: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df_interactions["interested_topic_id"] = user_info["interested_topic_id"] /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:71: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df_interactions['view_time'] = 0 /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:18: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df_interactions['updated_at'] = pd.to_datetime(df_interactions['updated_at']) /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:21: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy lambda x: (pd.Timestamp.now() - x['updated_at']).days + 1, axis=1) /home/ai/Documents/AI/gh-ai/recommendation/data_manager/deep_data_manager.py:23: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy df_interactions['card_recency'] = 1 / df_interactions['card_recency'] /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:4167: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy errors=errors, /home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/pandas/core/frame.py:3065: SettingWithCopyWarning: A value is trying to be set on a copy of a slice from a DataFrame. Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy self[k1] = value[k2] Traceback (most recent call last): File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 82, in score_phase() File "/home/ai/Documents/AI/gh-ai/recommendation/models/recommend_model_wrapper.py", line 76, in scorephase results = recommender.predict(predict_dataset) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 130, in _method_wrapper return method(self, *args, kwargs) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py", line 1599, in predict tmp_batch_outputs = predict_function(iterator) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 780, in call result = self._call(*args, *kwds) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 823, in _call self._initialize(args, kwds, add_initializers_to=initializers) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 697, in _initialize args, kwds)) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 2855, in _get_concrete_function_internal_garbage_collected graphfunction, , _ = self._maybe_define_function(args, kwargs) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3213, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py", line 3075, in _create_graph_function capture_by_value=self._capture_by_value), File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 986, in func_graph_from_py_func func_outputs = python_func(*func_args, *func_kwargs) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py", line 600, in wrapped_fn return weak_wrapped_fn().wrapped(args, **kwds) File "/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py", line 973, in wrapper raise e.ag_error_metadata.to_exception(e) ValueError: in user code:

/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1462 predict_function  *
    return step_function(self, iterator)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1452 step_function  **
    outputs = model.distribute_strategy.run(run_step, args=(data,))
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
    return fn(*args, **kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1445 run_step  **
    outputs = model.predict_step(data)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1418 predict_step
    return self(x, training=False)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py:985 __call__
    outputs = call_fn(inputs, *args, **kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:71 return_outputs_and_add_losses
    outputs, losses = fn(inputs, *args, **kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 wrap_with_training_arg
    lambda: replace_training_and_call(False))
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py:65 smart_cond
    pred, true_fn=true_fn, false_fn=false_fn, name=name)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py:56 smart_cond
    return false_fn()
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:170 <lambda>
    lambda: replace_training_and_call(False))
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/keras/saving/saved_model/utils.py:165 replace_training_and_call
    return wrapped_call(*args, **kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:780 __call__
    result = self._call(*args, **kwds)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:823 _call
    self._initialize(args, kwds, add_initializers_to=initializers)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:697 _initialize
    *args, **kwds))
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:2855 _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3213 _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/function.py:3075 _create_graph_function
    capture_by_value=self._capture_by_value),
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/eager/def_function.py:600 wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
/home/ai/Documents/AI/gh-ai/eurekai/lib/python3.6/site-packages/tensorflow/python/saved_model/function_deserialization.py:257 restored_function_body
    "\n\n".join(signature_descriptions)))

ValueError: Could not find matching function to call loaded from the SavedModel. Got:
  Positional arguments (2 total):
    * {'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, 'card_title': <tf.Tensor 'inputs_2:0' shape=(None, 1) dtype=string>, 'card_topic_id': <tf.Tensor 'inputs_3:0' shape=(None, 1) dtype=string>, 'user_id': <tf.Tensor 'inputs_5:0' shape=(None, 1) dtype=string>, 'interested_topic_id': <tf.Tensor 'inputs_4:0' shape=(None, 1) dtype=string>, 'view_time': <tf.Tensor 'inputs_6:0' shape=(None, 1) dtype=int32>, 'card_recency': <tf.Tensor 'inputs_1:0' shape=(None, 1) dtype=float64>}
    * False
  Keyword arguments: {}

Expected these arguments to match one of the following 4 option(s):

Option 1:
  Positional arguments (2 total):
    * {'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency')}
    * True
  Keyword arguments: {}

Option 2:
  Positional arguments (2 total):
    * {'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id')}
    * False
  Keyword arguments: {}

Option 3:
  Positional arguments (2 total):
    * {'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/interested_topic_id'), 'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='inputs/card_recency'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='inputs/view_time'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_title'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/user_id')}
    * False
  Keyword arguments: {}

Option 4:
  Positional arguments (2 total):
    * {'card_recency': TensorSpec(shape=(None,), dtype=tf.float64, name='card_recency'), 'interested_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='interested_topic_id'), 'card_topic_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_topic_id'), 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='card_id'), 'card_title': TensorSpec(shape=(None,), dtype=tf.string, name='card_title'), 'view_time': TensorSpec(shape=(None,), dtype=tf.int32, name='view_time'), 'user_id': TensorSpec(shape=(None,), dtype=tf.string, name='user_id')}
    * True
  Keyword arguments: {}

Process finished with exit code 1

[nttgithubntt]

Hj @maciejkula , I had find the cause of the problem, due to the mismatch shape between 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, and requirement input 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'). I have fixed it, tks you again :)

[yoke2c]

Hj @maciejkula , I had find the cause of the problem, due to the mismatch shape between 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, and requirement input 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id'). I have fixed it, tks you again :)

Hi @nttgithubntt I encountered the same error as you while trying to call model.predict or model.evaluate after saving the model using model.save and reloading the model again using tf.keras.models.load_model and understand that you have managed to fix it. Perhaps could you share or advise where the shape mismatch can be fixed? Thank you.

[nttgithubntt]

Hj @yoke2c , you push the log to check more detail. the common cause of this error is the tensor variable shape in training phase does not match the evaluating or predicting phase. I my case error: In training phase, the tensor variable 'card_id': <tf.Tensor 'inputs:0' shape=(None, 1) dtype=string>, has shape = shape=(None, 1) But in predicting phase, we input it with 'card_id': TensorSpec(shape=(None,), dtype=tf.string, name='inputs/card_id') has shape =(None,) Clearly, shape=(None, 1) not match shape =(None,).

[yoke2c]

Hi @nttgithubntt thank you for sharing! While I'm still trying to troubleshoot the shape issue, a work-around now is using model.save_weights() and then using model.load_weights() after re-creating the model.

[maciejkula]

I suspect it might be related to different batching that you're doing between training/evaluation and prediction.

[yoke2c]

I suspect it might be related to different batching that you're doing between training/evaluation and prediction.

Thanks for the advice @maciejkula, shall explore this as well.

[cory1219]

Hello @nttgithubntt, can you explain how you changed tensor variable shape from (None, 1) to (None,) thank!