search

tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
186.45k stars 74.32k forks source link

No gradient defined error when training Sequential model restored from SavedModel with `tf.keras.models.load_model` #34211

Closed yanndupis closed 4 years ago

yanndupis commented 5 years ago

System information

Describe the current behavior When training a Sequential model (Embedding + LSTM + Dense layer) restored from SavedModel with load_model(model = tf.keras.models.load_model('imodel_saved'), I am getting the following error : LookupError: No gradient defined for operation 'while' (op type: While).

I am able to train the same model, if the model was previously saved in h5 format (tf.keras.models.save_model(model, 'imdb_model', include_optimizer=True, save_format='h5 ). However it should work with `save_format='tf`` as well. It works for other models (e.g. convolutional model) but not for this specific model:

model = Sequential()
model.add(Embedding(20000, 128))
model.add(LSTM(128))
model.add(Dense(1, activation='sigmoid'))

Describe the expected behavior We should be able to train the model when restored from SavedModel.

Code to reproduce the issue

First run this script to define and save the Sequential model:

import numpy as np

import tensorflow as tf

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Embedding
from tensorflow.keras.layers import LSTM

model = Sequential()
model.add(Embedding(20000, 128))
model.add(LSTM(128))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

tf.keras.models.save_model(model, 
                          'imdb_model', 
                           include_optimizer=True, 
                           save_format='tf')

Then run the following script to restore the model and train it on the IMDB dataset:

import tensorflow as tf

from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence

(x_train, y_train), _ = imdb.load_data(num_words=20000)
x_train = sequence.pad_sequences(x_train, maxlen=80)

model = tf.keras.models.load_model('imdb_model')

model.fit(x_train, y_train, epochs=1)

Other info / logs Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached.


    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 582, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 629, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 619, in _backprop_function
    src_graph=self._func_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 715, in _registered_grad_fn
    return self._rewrite_forward_and_call_backward(op, *doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 661, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 582, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 629, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 619, in _backprop_function
    src_graph=self._func_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 637, in _GradientsHelper
    (op.name, op.type))
LookupError: No gradient defined for operation 'while' (op type: While)
 yanndupis@yanns-MBP  ~/Documents/dropoutlabs/project-cowbay/example/medical-symptoms   model-training ● ? ⍟1 
 yanndupis@yanns-MBP  ~/Documents/dropoutlabs/project-cowbay/example/medical-symptoms   model-training ● ? ⍟1  clear
 yanndupis@yanns-MBP  ~/Documents/dropoutlabs/project-cowbay/example/medical-symptoms   model-training ● ? ⍟1  python train_model.py

2019-11-12 17:36:48.074011: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-12 17:36:48.086305: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7ff129c9cae0 executing computations on platform Host. Devices:
2019-11-12 17:36:48.086321: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Host, Default Version
Train on 25000 samples
   32/25000 [..............................] - ETA: 1:28Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 2383, in get_attr
    c_api.TF_OperationGetAttrValueProto(self._c_op, name, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 345, in _MaybeCompile
    xla_compile = op.get_attr("_XlaCompile")
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 2387, in get_attr
    raise ValueError(str(e))
ValueError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 2383, in get_attr
    c_api.TF_OperationGetAttrValueProto(self._c_op, name, buf)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 345, in _MaybeCompile
    xla_compile = op.get_attr("_XlaCompile")
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 2387, in get_attr
    raise ValueError(str(e))
ValueError: Operation 'StatefulPartitionedCall' has no attr named '_XlaCompile'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 621, in _GradientsHelper
    grad_fn = ops.get_gradient_function(op)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 2541, in get_gradient_function
    return _gradient_registry.lookup(op_type)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/registry.py", line 97, in lookup
    "%s registry has no entry for: %s" % (self._name, name))
LookupError: gradient registry has no entry for: While

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train_model.py", line 11, in <module>
    model.fit(x_train, y_train, epochs=1)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training.py", line 728, in fit
    use_multiprocessing=use_multiprocessing)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 324, in fit
    total_epochs=epochs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2.py", line 123, in run_one_epoch
    batch_outs = execution_function(iterator)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 86, in execution_function
    distributed_function(input_fn))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 457, in __call__
    result = self._call(*args, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 503, in _call
    self._initialize(args, kwds, add_initializers_to=initializer_map)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 408, in _initialize
    *args, **kwds))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 1848, in _get_concrete_function_internal_garbage_collected
    graph_function, _, _ = self._maybe_define_function(args, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2150, in _maybe_define_function
    graph_function = self._create_graph_function(args, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 2041, in _create_graph_function
    capture_by_value=self._capture_by_value),
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/def_function.py", line 358, in wrapped_fn
    return weak_wrapped_fn().__wrapped__(*args, **kwds)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 73, in distributed_function
    per_replica_function, args=(model, x, y, sample_weights))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 760, in experimental_run_v2
    return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 1787, in call_for_each_replica
    return self._call_for_each_replica(fn, args, kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/distribute/distribute_lib.py", line 2132, in _call_for_each_replica
    return fn(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/autograph/impl/api.py", line 292, in wrapper
    return func(*args, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_v2_utils.py", line 264, in train_on_batch
    output_loss_metrics=model._output_loss_metrics)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 311, in train_on_batch
    output_loss_metrics=output_loss_metrics))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/training_eager.py", line 268, in _process_single_batch
    grads = tape.gradient(scaled_total_loss, trainable_weights)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/backprop.py", line 1014, in gradient
    unconnected_gradients=unconnected_gradients)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/imperative_grad.py", line 76, in imperative_grad
    compat.as_str(unconnected_gradients.value))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 738, in _backward_function
    return self._rewrite_forward_and_call_backward(call_op, *args)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 661, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 582, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 629, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 619, in _backprop_function
    src_graph=self._func_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 715, in _registered_grad_fn
    return self._rewrite_forward_and_call_backward(op, *doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 661, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 582, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 629, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 619, in _backprop_function
    src_graph=self._func_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
    return grad_fn()  # Exit early
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in <lambda>
    lambda: grad_fn(op, *out_grads))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 715, in _registered_grad_fn
    return self._rewrite_forward_and_call_backward(op, *doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 661, in _rewrite_forward_and_call_backward
    forward_function, backwards_function = self.forward_backward(len(doutputs))
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 582, in forward_backward
    forward, backward = self._construct_forward_backward(num_doutputs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 629, in _construct_forward_backward
    func_graph=backwards_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/framework/func_graph.py", line 915, in func_graph_from_py_func
    func_outputs = python_func(*func_args, **func_kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/eager/function.py", line 619, in _backprop_function
    src_graph=self._func_graph)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tensorflow_core/python/ops/gradients_util.py", line 637, in _GradientsHelper
    (op.name, op.type))
LookupError: No gradient defined for operation 'while' (op type: While)
ravikyram commented 5 years ago

@yanndupis

I have tried on colab with TF version 2.0 and i am not seeing any error message. Please, find the gist here.Thanks!

yanndupis commented 5 years ago

Hello @ravikyram , thank you very much for your quick response.

I should have been more precise. You will get the error, if you load the model in a different script (where the model hasn't been explicitly defined) then start training it. In the gist, the entire code is in the same Colab. You should be able to reproduce if you have two different Colabs.

In the first Colab you run:

import numpy as np

import tensorflow as tf

from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense,Embedding
from tensorflow.keras.layers import LSTM

model = Sequential()
model.add(Embedding(20000, 128))
model.add(LSTM(128))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

tf.keras.models.save_model(model, 
                          'imdb_model', 
                           include_optimizer=True, 
                           save_format='tf')

and in a different Colab you run:

import tensorflow as tf

from tensorflow.keras.datasets import imdb
from tensorflow.keras.preprocessing import sequence

(x_train, y_train), _ = imdb.load_data(num_words=20000)
x_train = sequence.pad_sequences(x_train, maxlen=80)

model = tf.keras.models.load_model('imdb_model')

model.fit(x_train, y_train, epochs=1)

Let me know if you have any question. Thank you!

ravikyram commented 5 years ago

I have tried on jupyter notebook with TF version 2.0 and was able to reproduce the issue.Please, find the files in attachment.Thanks! error.tar.gz

diNatale commented 4 years ago

Hi @yanndupis, @ravikyram , It looks like we have the same problem, with a similar error, when saving a model, and then loading it again, e.g. for retrain.

We save the model using tf.saved_model.save() and not Keras.

We are running on Ubuntu 16.04, Python 3.6, TF2.0 We have just tried and the problem still exists with TF2.1.0-rc1

ujjwal-ai commented 4 years ago

Is there any update on this ???

robert-moore commented 4 years ago

Hitting the same error with a bidirectional GRU layer

brresnic commented 4 years ago

hitting a similar error with an LSTM model

tlawren commented 4 years ago

Hitting similar issue with simple LSTM.

jvishnuvardhan commented 4 years ago

@yanndupis I cannot reproduce the issue with recent tf-nightly. Can you please check the gist here. I ran the first cell to save the model, then restarted colab's runtime, then ran second cell to load the model and execute model.fit.

Please close the issue if this was resolved for you. Thanks!

jvishnuvardhan commented 4 years ago

@yanndupis Closing this as it was resolved in tf-nightly. Please feel free to reopen if this was not resolved for you. Thanks!

tensorflow-bot[bot] commented 4 years ago

Are you satisfied with the resolution of your issue? Yes No

Gentatsu commented 4 years ago

Error still present in tf-nightly. Ran the example given by @jvishnuvardhan in two separate colab runtimes (saving in one, and loading in the other) and the error is still present:



    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:806 train_function  *
        return step_function(self, iterator)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:796 step_function  **
        outputs = model.distribute_strategy.run(run_step, args=(data,))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:1211 run
        return self._extended.call_for_each_replica(fn, args=args, kwargs=kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2585 call_for_each_replica
        return self._call_for_each_replica(fn, args, kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/distribute/distribute_lib.py:2945 _call_for_each_replica
        return fn(*args, **kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:789 run_step  **
        outputs = model.train_step(data)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:757 train_step
        self.trainable_variables)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/training.py:2722 _minimize
        gradients = tape.gradient(loss, trainable_variables)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/backprop.py:1073 gradient
        unconnected_gradients=unconnected_gradients)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/imperative_grad.py:77 imperative_grad
        compat.as_str(unconnected_gradients.value))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:797 _backward_function
        return self._rewrite_forward_and_call_backward(call_op, *args)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:712 _rewrite_forward_and_call_backward
        forward_function, backwards_function = self.forward_backward(len(doutputs))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:621 forward_backward
        forward, backward = self._construct_forward_backward(num_doutputs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:669 _construct_forward_backward
        func_graph=backwards_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:659 _backprop_function
        src_graph=self._func_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:669 _GradientsHelper
        lambda: grad_fn(op, *out_grads))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:336 _MaybeCompile
        return grad_fn()  # Exit early
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:669 <lambda>
        lambda: grad_fn(op, *out_grads))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:712 _rewrite_forward_and_call_backward
        forward_function, backwards_function = self.forward_backward(len(doutputs))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:621 forward_backward
        forward, backward = self._construct_forward_backward(num_doutputs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:669 _construct_forward_backward
        func_graph=backwards_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:659 _backprop_function
        src_graph=self._func_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:669 _GradientsHelper
        lambda: grad_fn(op, *out_grads))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:336 _MaybeCompile
        return grad_fn()  # Exit early
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:669 <lambda>
        lambda: grad_fn(op, *out_grads))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:712 _rewrite_forward_and_call_backward
        forward_function, backwards_function = self.forward_backward(len(doutputs))
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:621 forward_backward
        forward, backward = self._construct_forward_backward(num_doutputs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:669 _construct_forward_backward
        func_graph=backwards_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/func_graph.py:986 func_graph_from_py_func
        func_outputs = python_func(*func_args, **func_kwargs)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/function.py:659 _backprop_function
        src_graph=self._func_graph)
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_util.py:623 _GradientsHelper
        (op.name, op.type))

    LookupError: No gradient defined for operation 'while' (op type: StatelessWhile)```
jvishnuvardhan commented 4 years ago

@Gentatsu I ran the first part and then restarted the runtime and then ran the second part (loading). I don't see any issue. Please note that I ran both the code in one colab but two runtimes.

As this is an old issue, Can you please open a new issue with standalone code to reproduce the issue. Thanks!

ccffccffcc commented 4 years ago

Facing the same error with tensorflow 2.3.1 and python 3.8. The model includes an LSTM layer. And I want to load the model and retrain it. I have been using the same script. I use tf.save_model.save and tf.save_model.load.

in compute_loss(data, model, y_truth, inverse_transform) 1 def compute_loss(data,model,y_truth,inverse_transform): ----> 2 predictions,_,_,_ = model(data) 3 predictions=tf.matmul(predictions,inverse_transform) 4 loss = tf.math.reduce_mean((predictions-y_truth)**2) 5 return loss ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/saved_model/load.py in _call_attribute(instance, *args, **kwargs) 507 508 def _call_attribute(instance, *args, **kwargs): --> 509 return instance.__call__(*args, **kwargs) 510 511 ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in __call__(self, *args, **kwds) 778 else: 779 compiler = "nonXla" --> 780 result = self._call(*args, **kwds) 781 782 new_tracing_count = self._get_tracing_count() ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/def_function.py in _call(self, *args, **kwds) 812 # In this case we have not created variables on the first call. So we can 813 # run the first trace but we should fail if variables are created. --> 814 results = self._stateful_fn(*args, **kwds) 815 if self._created_variables: 816 raise ValueError("Creating variables on a non-first call to a function" ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs) 2827 with self._lock: 2828 graph_function, args, kwargs = self._maybe_define_function(args, kwargs) -> 2829 return graph_function._filtered_call(args, kwargs) # pylint: disable=protected-access 2830 2831 @property ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _filtered_call(self, args, kwargs, cancellation_manager) 1841 `args` and `kwargs`. 1842 """ -> 1843 return self._call_flat( 1844 [t for t in nest.flatten((args, kwargs), expand_composites=True) 1845 if isinstance(t, (ops.Tensor, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager) 1927 possible_gradient_type, 1928 executing_eagerly) -> 1929 forward_function, args_with_tangents = forward_backward.forward() 1930 if executing_eagerly: 1931 flat_outputs = forward_function.call( ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in forward(self) 1430 def forward(self): 1431 """Builds or retrieves a forward function for this call.""" -> 1432 forward_function = self._functions.forward( 1433 self._inference_args, self._input_tangents) 1434 return forward_function, self._inference_args + self._input_tangents ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in forward(self, inference_args, input_tangents) 1187 (self._forward, self._forward_graph, self._backward, 1188 self._forwardprop_output_indices, self._num_forwardprop_outputs) = ( -> 1189 self._forward_and_backward_functions(inference_args, input_tangents)) 1190 return self._forward 1191 ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _forward_and_backward_functions(self, inference_args, input_tangents) 1386 while len(outputs) < len(self._func_graph.outputs): 1387 outputs = list(self._func_graph.outputs) -> 1388 self._build_functions_for_outputs( 1389 outputs, inference_args, input_tangents) 1390 (forward_function, forward_graph, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _build_functions_for_outputs(self, outputs, inference_args, input_tangents) 893 graph_placeholder(gradient_dtype, gradient_shape)) 894 with ops.device(None): --> 895 gradients_wrt_inputs = gradients_util._GradientsHelper( # pylint: disable=protected-access 896 trainable_outputs, 897 self._func_graph.inputs, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph) 666 # If grad_fn was found, do not use SymbolicGradient even for 667 # functions. --> 668 in_grads = _MaybeCompile(grad_scope, op, func_call, 669 lambda: grad_fn(op, *out_grads)) 670 else: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _MaybeCompile(scope, op, func, grad_fn) 334 xla_scope = op.get_attr("_XlaScope").decode() 335 except ValueError: --> 336 return grad_fn() # Exit early 337 338 if not xla_compile: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in () 667 # functions. 668 in_grads = _MaybeCompile(grad_scope, op, func_call, --> 669 lambda: grad_fn(op, *out_grads)) 670 else: 671 # For function call ops, we add a 'SymbolicGradient' ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _rewrite_forward_and_call_backward(self, op, *doutputs) 710 def _rewrite_forward_and_call_backward(self, op, *doutputs): 711 """Add outputs to the forward call and feed them to the grad function.""" --> 712 forward_function, backwards_function = self.forward_backward(len(doutputs)) 713 if not backwards_function.outputs: 714 return backwards_function.structured_outputs ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in forward_backward(self, num_doutputs) 619 if forward_backward is not None: 620 return forward_backward --> 621 forward, backward = self._construct_forward_backward(num_doutputs) 622 self._cached_function_pairs[num_doutputs] = (forward, backward) 623 return forward, backward ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _construct_forward_backward(self, num_doutputs) 662 backwards_graph = func_graph_module.FuncGraph( 663 _backward_name(self._func_graph.name)) --> 664 func_graph_module.func_graph_from_py_func( 665 name=backwards_graph.name, 666 python_func=_backprop_function, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes) 984 _, original_func = tf_decorator.unwrap(python_func) 985 --> 986 func_outputs = python_func(*func_args, **func_kwargs) 987 988 # invariant: `func_outputs` contains only Tensors, CompositeTensors, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _backprop_function(*grad_ys) 653 def _backprop_function(*grad_ys): 654 with ops.device(None): --> 655 return gradients_util._GradientsHelper( # pylint: disable=protected-access 656 trainable_outputs, 657 self._func_graph.inputs, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph) 666 # If grad_fn was found, do not use SymbolicGradient even for 667 # functions. --> 668 in_grads = _MaybeCompile(grad_scope, op, func_call, 669 lambda: grad_fn(op, *out_grads)) 670 else: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _MaybeCompile(scope, op, func, grad_fn) 334 xla_scope = op.get_attr("_XlaScope").decode() 335 except ValueError: --> 336 return grad_fn() # Exit early 337 338 if not xla_compile: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in () 667 # functions. 668 in_grads = _MaybeCompile(grad_scope, op, func_call, --> 669 lambda: grad_fn(op, *out_grads)) 670 else: 671 # For function call ops, we add a 'SymbolicGradient' ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _rewrite_forward_and_call_backward(self, op, *doutputs) 710 def _rewrite_forward_and_call_backward(self, op, *doutputs): 711 """Add outputs to the forward call and feed them to the grad function.""" --> 712 forward_function, backwards_function = self.forward_backward(len(doutputs)) 713 if not backwards_function.outputs: 714 return backwards_function.structured_outputs ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in forward_backward(self, num_doutputs) 619 if forward_backward is not None: 620 return forward_backward --> 621 forward, backward = self._construct_forward_backward(num_doutputs) 622 self._cached_function_pairs[num_doutputs] = (forward, backward) 623 return forward, backward ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _construct_forward_backward(self, num_doutputs) 662 backwards_graph = func_graph_module.FuncGraph( 663 _backward_name(self._func_graph.name)) --> 664 func_graph_module.func_graph_from_py_func( 665 name=backwards_graph.name, 666 python_func=_backprop_function, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes) 984 _, original_func = tf_decorator.unwrap(python_func) 985 --> 986 func_outputs = python_func(*func_args, **func_kwargs) 987 988 # invariant: `func_outputs` contains only Tensors, CompositeTensors, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _backprop_function(*grad_ys) 653 def _backprop_function(*grad_ys): 654 with ops.device(None): --> 655 return gradients_util._GradientsHelper( # pylint: disable=protected-access 656 trainable_outputs, 657 self._func_graph.inputs, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph) 666 # If grad_fn was found, do not use SymbolicGradient even for 667 # functions. --> 668 in_grads = _MaybeCompile(grad_scope, op, func_call, 669 lambda: grad_fn(op, *out_grads)) 670 else: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _MaybeCompile(scope, op, func, grad_fn) 334 xla_scope = op.get_attr("_XlaScope").decode() 335 except ValueError: --> 336 return grad_fn() # Exit early 337 338 if not xla_compile: ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in () 667 # functions. 668 in_grads = _MaybeCompile(grad_scope, op, func_call, --> 669 lambda: grad_fn(op, *out_grads)) 670 else: 671 # For function call ops, we add a 'SymbolicGradient' ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _rewrite_forward_and_call_backward(self, op, *doutputs) 710 def _rewrite_forward_and_call_backward(self, op, *doutputs): 711 """Add outputs to the forward call and feed them to the grad function.""" --> 712 forward_function, backwards_function = self.forward_backward(len(doutputs)) 713 if not backwards_function.outputs: 714 return backwards_function.structured_outputs ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in forward_backward(self, num_doutputs) 619 if forward_backward is not None: 620 return forward_backward --> 621 forward, backward = self._construct_forward_backward(num_doutputs) 622 self._cached_function_pairs[num_doutputs] = (forward, backward) 623 return forward, backward ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _construct_forward_backward(self, num_doutputs) 662 backwards_graph = func_graph_module.FuncGraph( 663 _backward_name(self._func_graph.name)) --> 664 func_graph_module.func_graph_from_py_func( 665 name=backwards_graph.name, 666 python_func=_backprop_function, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/framework/func_graph.py in func_graph_from_py_func(name, python_func, args, kwargs, signature, func_graph, autograph, autograph_options, add_control_dependencies, arg_names, op_return_value, collections, capture_by_value, override_flat_arg_shapes) 984 _, original_func = tf_decorator.unwrap(python_func) 985 --> 986 func_outputs = python_func(*func_args, **func_kwargs) 987 988 # invariant: `func_outputs` contains only Tensors, CompositeTensors, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/eager/function.py in _backprop_function(*grad_ys) 653 def _backprop_function(*grad_ys): 654 with ops.device(None): --> 655 return gradients_util._GradientsHelper( # pylint: disable=protected-access 656 trainable_outputs, 657 self._func_graph.inputs, ~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/tensorflow/python/ops/gradients_util.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph) 619 grad_fn = func_call.python_grad_func 620 else: --> 621 raise LookupError( 622 "No gradient defined for operation '%s' (op type: %s)" % 623 (op.name, op.type)) LookupError: No gradient defined for operation 'while' (op type: While)