search

tensorflow / tensorflow

An Open Source Machine Learning Framework for Everyone
https://tensorflow.org
Apache License 2.0
186.25k stars 74.3k forks source link

TypeError: this __dict__ descriptor does not support '_DictWrapper' objects during trivial model save #63548

Open iamthebot opened 7 months ago

iamthebot commented 7 months ago

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No (bug doesn't exist in tf-nightly 2.17.0.dev20240312)

Source

source

TensorFlow version

v2.16.1

Custom code

Yes

OS platform and distribution

OSX

Mobile device

No response

Python version

3.12

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

When calling tf.saved_model.save(model, saved_model_path)

we see:

../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py:190: in list_children
    for name, child in super(_AugmentedGraphView, self).list_children(
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/graph_view.py:75: in list_children
    for name, ref in super(ObjectGraphView,
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/trackable_view.py:85: in children
    ref = converter.convert_to_trackable(ref, parent=obj)
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/trackable/converter.py:31: in convert_to_trackable
    if (tensor_util.is_tf_type(obj) and
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/framework/tensor_util.py:1156: in is_tf_type
    return isinstance(x, tf_type_classes)
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/typing.py:1871: in __instancecheck__
    val = getattr_static(instance, attr)
../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/inspect.py:1839: in getattr_static
    instance_result = _check_instance(obj, attr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = DictWrapper({'input_shape': [(None, 16), (None, 32)]}), attr = 'is_tensor_like'

    def _check_instance(obj, attr):
        instance_dict = {}
        try:
>           instance_dict = object.__getattribute__(obj, "__dict__")
E           TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

../../../../miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/inspect.py:1793: TypeError

I suspect this is related to #59869 which was supposedly fixed. However, in 2.16.1 TF removes the pin on wrapt and the issue indeed persists. I've even tried downgrading wrapt to 1.14.1 and the issue remains.

Standalone code to reproduce the issue

import tensorflow as tf
from tensorflow.keras import layers
import tempfile

def get_two_tower_models():
    online_features = [layers.Input(shape=(32,)), layers.Input(shape=(16,))]
    offline_features = [layers.Input(shape=(16,)), layers.Input(shape=(32,))]
    all_features = []
    all_features.extend(online_features)
    all_features.extend(offline_features)

    def get_offline_tower(offline_features):
        offline_inputs = layers.concatenate(offline_features, name="offline_concatenated")
        offline_hidden = layers.Dense(32, activation="tanh", name="offline_hidden_1")(offline_inputs)
        offline_hidden = layers.Dense(16, activation="tanh", name="offline_hidden_2")(offline_hidden)
        offline_final_embed = layers.Dense(8, name="offline_hidden_3")(offline_hidden)

        return offline_final_embed

    def get_online_tower(online_features):
        online_inputs = layers.concatenate(online_features, name="online_concatenated")
        online_hidden = layers.Dense(32, activation="tanh", name="online_hidden_1")(online_inputs)
        online_hidden = layers.Dense(16, activation="tanh", name="online_hidden_2")(online_hidden)
        online_final_embed = layers.Dense(8, name="online_hidden_3")(online_hidden)

        return online_final_embed

    offline_tower_embed = get_offline_tower(offline_features)
    online_tower_embed = get_online_tower(online_features)

    # We normalize vectors with L2 norm to make sure we get the cosine similarity
    offline_online_dot = layers.Dot(axes=1, normalize=True)([offline_tower_embed, online_tower_embed])

    offline_tower_model = tf.keras.Model(inputs=offline_features, outputs=offline_tower_embed)
    online_tower_model = tf.keras.Model(inputs=online_features, outputs=online_tower_embed)

    full_model = tf.keras.Model(inputs=all_features, outputs=offline_online_dot)

    return (full_model, offline_tower_model, online_tower_model)

full_model, offline_tower_model, online_tower_model = get_two_tower_models()

with tempfile.TemporaryDirectory() as tmpdirname:
    tf.saved_model.save(online_tower_model, tmpdirname)

### Relevant log output

```shell
Traceback (most recent call last):
  File "/Users/alfredo_luque/repos/git.musta.ch/airbnb/bighead-service/packages/ml-frameworks/tensorflow/tests/minimal_repro.py", line 44, in <module>
    tf.saved_model.save(online_tower_model, tmpdirname)
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 1392, in save
    save_and_return_nodes(obj, export_dir, signatures, options)
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 1427, in save_and_return_nodes
    _build_meta_graph(obj, signatures, options, meta_graph_def))
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 1642, in _build_meta_graph
    return _build_meta_graph_impl(obj, signatures, options, meta_graph_def)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 1564, in _build_meta_graph_impl
    saveable_view = _SaveableView(augmented_graph_view, options)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 285, in __init__
    checkpoint_util.objects_ids_and_slot_variables_and_paths(
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/util.py", line 160, in objects_ids_and_slot_variables_and_paths
    trackable_objects, node_paths = graph_view.breadth_first_traversal()
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/graph_view.py", line 124, in breadth_first_traversal
    return self._breadth_first_traversal()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 156, in _breadth_first_traversal
    super(_AugmentedGraphView, self)._breadth_first_traversal())
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/graph_view.py", line 128, in _breadth_first_traversal
    return super(ObjectGraphView, self)._descendants_with_paths()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/trackable_view.py", line 111, in _descendants_with_paths
    for name, dependency in self.children(current_trackable).items():
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/graph_view.py", line 97, in children
    for name, ref in self.list_children(obj, **kwargs):
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/saved_model/save.py", line 190, in list_children
    for name, child in super(_AugmentedGraphView, self).list_children(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/graph_view.py", line 75, in list_children
    for name, ref in super(ObjectGraphView,
                     ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/checkpoint/trackable_view.py", line 85, in children
    ref = converter.convert_to_trackable(ref, parent=obj)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/trackable/converter.py", line 31, in convert_to_trackable
    if (tensor_util.is_tf_type(obj) and
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/site-packages/tensorflow/python/framework/tensor_util.py", line 1156, in is_tf_type
    return isinstance(x, tf_type_classes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/typing.py", line 1871, in __instancecheck__
    val = getattr_static(instance, attr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/inspect.py", line 1839, in getattr_static
    instance_result = _check_instance(obj, attr)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/alfredo_luque/miniforge3/envs/local--bighead--v0.0.1/lib/python3.12/inspect.py", line 1793, in _check_instance
    instance_dict = object.__getattribute__(obj, "__dict__")
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: this __dict__ descriptor does not support '_DictWrapper' objects
iamthebot commented 7 months ago

I can verify this bug is not present in 2.15.1. Nor is it present in tf-nightly==2.17.0.dev20240312 with keras-nightly==3.1.0.dev2024031203

darrina commented 7 months ago

Same issue, even simpler example:

import tensorflow as tf
pretrained_model = tf.keras.applications.MobileNet()
tf.saved_model.save(pretrained_model, "mobilenet")
TypeError: this __dict__ descriptor does not support '_DictWrapper' objects

TensorFlow 2.16.0rc0 and 2.16.1 are the only tensorflow versions currently available using pip install.

txchen commented 7 months ago

I am having the exact same issue.

iamthebot commented 7 months ago

@darrina yeah the only workaround we have right now is tf-nightly to enable model saving w/ py3.12... but that's obviously not a viable workaround for production use.

victorface2 commented 7 months ago

Running into a similar issue.

Venkat6871 commented 7 months ago

Hi @iamthebot ,

Sorry for the delay, I tried to run your code on Colab using TF v2.15 and 2.16, But I don't faced any issue here. Could you check again. Please find the gist here for reference.

Thank you!

niko247 commented 7 months ago

@Venkat6871 I have similar issue and I think the issue is with python 3.12, with python 3.11 it works

jagiella commented 7 months ago

I confirm this to be linked to python 3.12.

I tested tensorflow 2.16, 2.16.1 and 2.17.0.dev20240317. There problem occured with all of them.

iamthebot commented 7 months ago

@Venkat6871 yep looks like Python 3.12 is required to reproduce this issue. Your collab is running Python 3.10.

jagiella commented 7 months ago

By the way disabling wrapt by setting an environment variable WRAPT_DISABLE_EXTENSIONS (as hinted by wrapt developer https://github.com/GrahamDumpleton/wrapt/issues/231#issuecomment-1455800902) will make the scripts run as expected:

WRAPT_DISABLE_EXTENSIONS=1 python3 python_script.py

It seems to be caused by a fix in the C extension of wrapt which was not adopted yet by tensorflow: https://github.com/GrahamDumpleton/wrapt/issues/231

zwikst commented 6 months ago

same issue for me...

import sys
print(sys.version_info)
sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0)
print(tf.__version__)
2.16.1
ash-sony commented 5 months ago

@Venkat6871 yep looks like Python 3.12 is required to reproduce this issue. Your collab is running Python 3.10.

it also happens using newer versions of typing_extensions on <3.12. Happened for me on 3.9 with typing_extensions=4.12, even on tensorflow 2.15 and wrapt < 1.15. Is this issue being looked into?

jdsalmonson commented 3 months ago

FYI, same issue here: python==3.12.4, tf==2.16.2 gives this error, but python==3.11.9, tf==2.15.0 works fine.

majic-ivan commented 3 months ago

Same issue with: TypeError: this dict descriptor does not support '_DictWrapper' objects

This happened when trying to follow the SAC agent tutorial on the tf agents website. Particularly, the "Learners" bit.

I use a custom environment, python == 3.10.12, tf == 2.14.0. wrapt ==1.14.1

parveenduhan commented 1 month ago

Facing issue with tf=2.17 (installed from source) and python 3.12.6 in tf.saved_model.save(model,path) typing-extensions:4.12.2, protobuf(python):4.21.9

I need to save model so that I can load(and use) it using cpp-api.

Oblynx commented 1 month ago

Do you know which is the minimum version of typing-extensions that doesn't cause this?