TypeError: Expected Operation, Variable, or Tensor, got None while saving tensorflow model

[jonanem]

🐛 Bug

Model I am using (Bert, XLNet....): TFAlbert

Language I am using the model on (English, Chinese....): English

The tasks I am working on is:

• [ ] an official GLUE/SQUaD task: (give the name): GLUE
• [ ] my own task or dataset: (give details)

Environment

• OS: Linux
• Python version: Python 3.7.5 /
• Tensorflow version: 2.0.0
• Using GPU ? GPU

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): Yes
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux
• TensorFlow installed from (source or binary): Source
• TensorFlow version: '2.0.0'
• Python version: Python 3.7.5 /Conda
• CUDA/cuDNN version: cuda10.0_0/cudnn-7.6.5
• GPU model and memory: Tesla V100-PCIE / 32 GB memory

Describe the current behavior I am getting TypeError: Expected Operation, Variable, or Tensor, got None while saving the model using model.save('../output/my_model')

---

TypeError Traceback (most recent call last)

in ----> 1 model.save('../output/my_model') /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/network.py in save(self, filepath, overwrite, include_optimizer, save_format, signatures, options) 973 """ 974 saving.save_model(self, filepath, overwrite, include_optimizer, save_format, --> 975 signatures, options) 976 977 def save_weights(self, filepath, overwrite=True, save_format=None): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/save.py in save_model(model, filepath, overwrite, include_optimizer, save_format, signatures, options) 113 else: 114 saved_model_save.save(model, filepath, overwrite, include_optimizer, --> 115 signatures, options) 116 117 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save.py in save(model, filepath, overwrite, include_optimizer, signatures, options) 72 # default learning phase placeholder. 73 with K.learning_phase_scope(0): ---> 74 save_lib.save(model, filepath, signatures, options) 75 76 if not include_optimizer: /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/save.py in save(obj, export_dir, signatures, options) 868 if signatures is None: 869 signatures = signature_serialization.find_function_to_export( --> 870 checkpoint_graph_view) 871 872 signatures = signature_serialization.canonicalize_signatures(signatures) /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/signature_serialization.py in find_function_to_export(saveable_view) 62 # If the user did not specify signatures, check the root object for a function 63 # that can be made into a signature. ---> 64 functions = saveable_view.list_functions(saveable_view.root) 65 signature = functions.get(DEFAULT_SIGNATURE_ATTR, None) 66 if signature is not None: /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/saved_model/save.py in list_functions(self, obj) 139 if obj_functions is None: 140 obj_functions = obj._list_functions_for_serialization( # pylint: disable=protected-access --> 141 self._serialization_cache) 142 self._functions[obj] = obj_functions 143 return obj_functions /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in _list_functions_for_serialization(self, serialization_cache) 2420 def _list_functions_for_serialization(self, serialization_cache): 2421 return (self._trackable_saved_model_saver -> 2422 .list_functions_for_serialization(serialization_cache)) 2423 2424 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/base_serialization.py in list_functions_for_serialization(self, serialization_cache) 89 `ConcreteFunction`. 90 """ ---> 91 fns = self.functions_to_serialize(serialization_cache) 92 93 # The parent AutoTrackable class saves all user-defined tf.functions, and /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/layer_serialization.py in functions_to_serialize(self, serialization_cache) 77 def functions_to_serialize(self, serialization_cache): 78 return (self._get_serialized_attributes( ---> 79 serialization_cache).functions_to_serialize) 80 81 def _get_serialized_attributes(self, serialization_cache): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/layer_serialization.py in _get_serialized_attributes(self, serialization_cache) 92 93 object_dict, function_dict = self._get_serialized_attributes_internal( ---> 94 serialization_cache) 95 96 serialized_attr.set_and_validate_objects(object_dict) /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/model_serialization.py in _get_serialized_attributes_internal(self, serialization_cache) 45 # cache (i.e. this is the root level object). 46 if len(serialization_cache[constants.KERAS_CACHE_KEY]) == 1: ---> 47 default_signature = save_impl.default_save_signature(self.obj) 48 49 # Other than the default signature function, all other attributes match with /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in default_save_signature(layer) 204 original_losses = _reset_layer_losses(layer) 205 fn = saving_utils.trace_model_call(layer) --> 206 fn.get_concrete_function() 207 _restore_layer_losses(original_losses) 208 return fn /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in get_concrete_function(self, *args, **kwargs) 774 if self._stateful_fn is None: 775 initializer_map = object_identity.ObjectIdentityDictionary() --> 776 self._initialize(args, kwargs, add_initializers_to=initializer_map) 777 self._initialize_uninitialized_variables(initializer_map) 778 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _initialize(self, args, kwds, add_initializers_to) 406 self._concrete_stateful_fn = ( 407 self._stateful_fn._get_concrete_function_internal_garbage_collected( # pylint: disable=protected-access --> 408 *args, **kwds)) 409 410 def invalid_creator_scope(*unused_args, **unused_kwds): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _get_concrete_function_internal_garbage_collected(self, *args, **kwargs) 1846 if self.input_signature: 1847 args, kwargs = None, None -> 1848 graph_function, _, _ = self._maybe_define_function(args, kwargs) 1849 return graph_function 1850 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs) 2148 graph_function = self._function_cache.primary.get(cache_key, None) 2149 if graph_function is None: -> 2150 graph_function = self._create_graph_function(args, kwargs) 2151 self._function_cache.primary[cache_key] = graph_function 2152 return graph_function, args, kwargs /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes) 2039 arg_names=arg_names, 2040 override_flat_arg_shapes=override_flat_arg_shapes, -> 2041 capture_by_value=self._capture_by_value), 2042 self._function_attributes, 2043 # Tell the ConcreteFunction to clean up its graph once it goes out of /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes) 913 converted_func) 914 --> 915 func_outputs = python_func(*func_args, **func_kwargs) 916 917 # invariant: `func_outputs` contains only Tensors, CompositeTensors, /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds) 356 # __wrapped__ allows AutoGraph to swap in a converted function. We give 357 # the function a weak reference to itself to avoid a reference cycle. --> 358 return weak_wrapped_fn().__wrapped__(*args, **kwds) 359 weak_wrapped_fn = weakref.ref(wrapped_fn) 360 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saving_utils.py in _wrapped_model(*args) 141 with base_layer_utils.call_context().enter( 142 model, inputs=inputs, build_graph=False, training=False, saving=True): --> 143 outputs_list = nest.flatten(model(inputs=inputs, training=False)) 144 145 try: /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs) 845 outputs = base_layer_utils.mark_as_return(outputs, acd) 846 else: --> 847 outputs = call_fn(cast_inputs, *args, **kwargs) 848 849 except errors.OperatorNotAllowedInGraphError as e: /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs) 290 def wrapper(*args, **kwargs): 291 with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED): --> 292 return func(*args, **kwargs) 293 294 if inspect.isfunction(func) or inspect.ismethod(func): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/transformers/modeling_tf_albert.py in call(self, inputs, **kwargs) 783 784 def call(self, inputs, **kwargs): --> 785 outputs = self.albert(inputs, **kwargs) 786 787 pooled_output = outputs[1] /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs) 845 outputs = base_layer_utils.mark_as_return(outputs, acd) 846 else: --> 847 outputs = call_fn(cast_inputs, *args, **kwargs) 848 849 except errors.OperatorNotAllowedInGraphError as e: /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py in wrapper(*args, **kwargs) 290 def wrapper(*args, **kwargs): 291 with ag_ctx.ControlStatusCtx(status=ag_ctx.Status.DISABLED): --> 292 return func(*args, **kwargs) 293 294 if inspect.isfunction(func) or inspect.ismethod(func): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/transformers/modeling_tf_albert.py in call(self, inputs, attention_mask, token_type_ids, position_ids, head_mask, inputs_embeds, training) 680 681 embedding_output = self.embeddings( --> 682 [input_ids, position_ids, token_type_ids, inputs_embeds], training=training) 683 encoder_outputs = self.encoder( 684 [embedding_output, extended_attention_mask, head_mask], training=training) /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in __call__(self, inputs, *args, **kwargs) 889 with base_layer_utils.autocast_context_manager( 890 self._compute_dtype): --> 891 outputs = self.call(cast_inputs, *args, **kwargs) 892 self._handle_activity_regularization(inputs, outputs) 893 self._set_mask_metadata(inputs, outputs, input_masks) /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in return_outputs_and_add_losses(*args, **kwargs) 55 inputs = args[inputs_arg_index] 56 args = args[inputs_arg_index + 1:] ---> 57 outputs, losses = fn(inputs, *args, **kwargs) 58 layer.add_loss(losses, inputs) 59 return outputs /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs) 109 training, 110 lambda: replace_training_and_call(True), --> 111 lambda: replace_training_and_call(False)) 112 113 # Create arg spec for decorated function. If 'training' is not defined in the /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in smart_cond(pred, true_fn, false_fn, name) 57 pred, true_fn=true_fn, false_fn=false_fn, name=name) 58 return smart_module.smart_cond( ---> 59 pred, true_fn=true_fn, false_fn=false_fn, name=name) 60 61 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name) 54 return true_fn() 55 else: ---> 56 return false_fn() 57 else: 58 return control_flow_ops.cond(pred, true_fn=true_fn, false_fn=false_fn, /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in () 109 training, 110 lambda: replace_training_and_call(True), --> 111 lambda: replace_training_and_call(False)) 112 113 # Create arg spec for decorated function. If 'training' is not defined in the /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in replace_training_and_call(training) 104 def replace_training_and_call(training): 105 set_training_arg(training, training_arg_index, args, kwargs) --> 106 return wrapped_call(*args, **kwargs) 107 108 return tf_utils.smart_cond( /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in __call__(self, *args, **kwargs) 531 if not self.call_collection.tracing: 532 self.call_collection.add_trace(*args, **kwargs) --> 533 return super(LayerCall, self).__call__(*args, **kwargs) 534 535 def get_concrete_function(self, *args, **kwargs): /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in __call__(self, *args, **kwds) 455 456 tracing_count = self._get_tracing_count() --> 457 result = self._call(*args, **kwds) 458 if tracing_count == self._get_tracing_count(): 459 self._call_counter.called_without_tracing() /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in _call(self, *args, **kwds) 492 # In this case we have not created variables on the first call. So we can 493 # run the first trace but we should fail if variables are created. --> 494 results = self._stateful_fn(*args, **kwds) 495 if self._created_variables: 496 raise ValueError("Creating variables on a non-first call to a function" /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in __call__(self, *args, **kwargs) 1820 def __call__(self, *args, **kwargs): 1821 """Calls a graph function specialized to the inputs.""" -> 1822 graph_function, args, kwargs = self._maybe_define_function(args, kwargs) 1823 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access 1824 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _maybe_define_function(self, args, kwargs) 2148 graph_function = self._function_cache.primary.get(cache_key, None) 2149 if graph_function is None: -> 2150 graph_function = self._create_graph_function(args, kwargs) 2151 self._function_cache.primary[cache_key] = graph_function 2152 return graph_function, args, kwargs /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py in _create_graph_function(self, args, kwargs, override_flat_arg_shapes) 2039 arg_names=arg_names, 2040 override_flat_arg_shapes=override_flat_arg_shapes, -> 2041 capture_by_value=self._capture_by_value), 2042 self._function_attributes, 2043 # Tell the ConcreteFunction to clean up its graph once it goes out of /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes) 913 converted_func) 914 --> 915 func_outputs = python_func(*func_args, **func_kwargs) 916 917 # invariant: `func_outputs` contains only Tensors, CompositeTensors, /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py in wrapped_fn(*args, **kwds) 356 # __wrapped__ allows AutoGraph to swap in a converted function. We give 357 # the function a weak reference to itself to avoid a reference cycle. --> 358 return weak_wrapped_fn().__wrapped__(*args, **kwds) 359 weak_wrapped_fn = weakref.ref(wrapped_fn) 360 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in wrapper(*args, **kwargs) 513 layer, inputs=inputs, build_graph=False, training=training, 514 saving=True): --> 515 ret = method(*args, **kwargs) 516 _restore_layer_losses(original_losses) 517 return ret /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in wrap_with_training_arg(*args, **kwargs) 109 training, 110 lambda: replace_training_and_call(True), --> 111 lambda: replace_training_and_call(False)) 112 113 # Create arg spec for decorated function. If 'training' is not defined in the /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in smart_cond(pred, true_fn, false_fn, name) 57 pred, true_fn=true_fn, false_fn=false_fn, name=name) 58 return smart_module.smart_cond( ---> 59 pred, true_fn=true_fn, false_fn=false_fn, name=name) 60 61 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/framework/smart_cond.py in smart_cond(pred, true_fn, false_fn, name) 54 return true_fn() 55 else: ---> 56 return false_fn() 57 else: 58 return control_flow_ops.cond(pred, true_fn=true_fn, false_fn=false_fn, /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in () 109 training, 110 lambda: replace_training_and_call(True), --> 111 lambda: replace_training_and_call(False)) 112 113 # Create arg spec for decorated function. If 'training' is not defined in the /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/utils.py in replace_training_and_call(training) 104 def replace_training_and_call(training): 105 set_training_arg(training, training_arg_index, args, kwargs) --> 106 return wrapped_call(*args, **kwargs) 107 108 return tf_utils.smart_cond( /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/saving/saved_model/save_impl.py in call_and_return_conditional_losses(inputs, *args, **kwargs) 555 layer_call = _get_layer_call_method(layer) 556 def call_and_return_conditional_losses(inputs, *args, **kwargs): --> 557 return layer_call(inputs, *args, **kwargs), layer.get_losses_for(inputs) 558 return _create_call_fn_decorator(layer, call_and_return_conditional_losses) 559 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py in get_losses_for(self, inputs) 1382 losses = [l for l in self.losses if not l._unconditional_loss] 1383 inputs = nest.flatten(inputs) -> 1384 reachable = tf_utils.get_reachable_from_inputs(inputs, losses) 1385 return [l for l in losses if l in reachable] 1386 /app/AI_RD/conda/envs/cont_tag_sup/lib/python3.7/site-packages/tensorflow_core/python/keras/utils/tf_utils.py in get_reachable_from_inputs(inputs, targets) 132 outputs = x.consumers() 133 else: --> 134 raise TypeError('Expected Operation, Variable, or Tensor, got ' + str(x)) 135 136 for y in outputs: TypeError: Expected Operation, Variable, or Tensor, got None **Describe the expected behavior** **Code to reproduce the issue** `import tensorflow as tf` `import pandas as pd` `from sklearn.model_selection import train_test_split` `import transformers` `from transformers import AlbertConfig` `from transformers import AlbertTokenizer` `from transformers import TFAlbertForSequenceClassification` `from transformers import glue_convert_examples_to_features` `data_df = pd.read_excel("../input/test.xlsx")` `model_dir = '../input/albert_xxlarge_v2/'` `EPOCHS = 3` `MAX_SEQ_LENGTH = 256` `label_list = [0,1]` `config = AlbertConfig.from_pretrained('albert-xxlarge-v2')` `tokenizer = AlbertTokenizer.from_pretrained('albert-xxlarge-v2', cache_dir=model_dir)` `model = TFAlbertForSequenceClassification.from_pretrained('albert-xxlarge-v2', ``cache_dir=model_dir, config=config)` `train_df, test_df = train_test_split(data_df[['id','text1', 'text2', 'LABEL']], random_state=42, shuffle=True, test_size=0.20, stratify=data_df['LABEL'])` `train_InputExamples = train_df.apply(lambda x: InputExample(guid=x['id'], text_a=x['text1'], text_b=x['text2'], label=x['LABEL']), axis=1)` `train_dataset = glue_convert_examples_to_features(examples=train_InputExamples, tokenizer=tokenizer, max_length=MAX_SEQ_LENGTH, label_list = label_list, output_mode="classification")` `optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=1e-08)` `loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)` `metric = tf.keras.metrics.SparseCategoricalAccuracy('accuracy')` `input_ids_train = []` `attention_mask_train = []` `token_type_ids_train = []` `output_label_train = []` `for f in train_dataset:` `input_ids_train.append(f.input_ids)` `attention_mask_train.append(f.attention_mask)` `token_type_ids_train.append(f.token_type_ids)` `output_label_train.append(f.label)` `model.compile(optimizer=optimizer, loss=loss, metrics=[metric])` `input_ids_train = np.array(input_ids_train)` `attention_mask_train = np.array(attention_mask_train)` `token_type_ids_train = np.array(token_type_ids_train)` `output_label_train = np.array(output_label_train)` `model.fit([input_ids_train,attention_mask_train, token_type_ids_train], y=output_label_train, epochs = EPOCHS, batch_size=4)` `model.save('../output/my_model')`

[jonanem]

The training of the model is successful, but getting errors only while saving the model

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.