lidatong / dataclasses-json

Easily serialize Data Classes to and from JSON
MIT License
1.34k stars 151 forks source link

[BUG] Incompatibility with from __future__ import annotations Affects Overriding of Global Config in Python 3.11 #458

Closed estyxx closed 10 months ago

estyxx commented 11 months ago

Description

I found a bug where the use of from __future__ import annotations causes dataclasses_json to fail when trying to override the global configuration for specific types. Since from __future__ import annotations changes the behaviour of annotations to be stored as string literals rather than evaluated expressions, the comparison of types in functions like _user_overrides_or_exts fails, leading to incorrect matching of types.

This issue particularly affects the overriding of the global config for encoders and decoders, and as a result, the library is not able to match the type correctly when type annotations are imported, leading to a failure in using the correct decoder method.

As a result I'm getting an exception:

Traceback (most recent call last):
  File "app.py", line 6, in <module>
    p = Page.from_dict(data)
  File ".../lib/python3.11/site-packages/dataclasses_json/api.py", line 70, in from_dict
    return _decode_dataclass(cls, kvs, infer_missing)
  File ".../lib/python3.11/site-packages/dataclasses_json/core.py", line 252, in _decode_dataclass
    init_kwargs[field.name] = _decode_generic(
  File ".../lib/python3.11/site-packages/dataclasses_json/core.py", line 356, in _decode_generic
    res = _support_extended_types(type_arg, value)
  File ".../lib/python3.11/site-packages/dataclasses_json/core.py", line 272, in _support_extended_types
    res = datetime.fromtimestamp(field_value, tz=tz)
TypeError: 'str' object cannot be interpreted as an integer

Can be related to #382

A possible solution could be using typing.get_type_hints instead.

Code snippet that reproduces the issue

from __future__ import annotations
from dataclasses_json import global_config
from dataclasses_json import LetterCase, dataclass_json
import dataclasses_json
from dataclasses import dataclass
import datetime
from typing import Optional

dataclasses_json.cfg.global_config.encoders[Optional[datetime.datetime]] = datetime.datetime.isoformat
dataclasses_json.cfg.global_config.decoders[Optional[datetime.datetime]] = datetime.datetime.fromisoformat

@dataclass_json(letter_case=LetterCase.CAMEL)
@dataclass
class Page:
    first_published_at: Optional[datetime.datetime]

data = {"first_published_at": "2023-07-24T14:03:57.546769+00:00"}
p = Page.from_dict(data)
print(p)  # Issue: The correct decoder method is not used due to the failure to match the types.

Describe the results you expected

Expected Behavior

The type annotations should be correctly matched with the encoders and decoders, whether or not from __future__ import annotations is used, allowing the correct overriding of the global config.

Actual Behavior

The comparison of types fails when from __future__ import annotations is used, causing the incorrect decoder method to be used; as a result, the conversation fails.

Python version you are using

Python 3.11.2

Environment description

name version location
mypy-extensions 1.0.0
typing-inspect 0.9.0
dataclasses-json 0.5.13
typing_extensions 4.7.1
marshmallow 3.20.1
packaging 23.1
marshmallow-enum 1.5.1
george-zubrienko commented 11 months ago

@matt035343 take a look? 🐱

USSX-Hares commented 11 months ago

As Python documentation states, __future__.annotations is a feature that will be removed in future releases. This is caused by the changed design direction/philosophy of the official Python community.

On the docs.python.org:

from __future__ import annotations was previously scheduled to become mandatory in Python 3.10, but the Python Steering Council twice decided to delay the change (announcement for Python 3.10; announcement for Python 3.11). No final decision has been made yet. See also PEP 563 and PEP 649.

PEP-649:

If accepted (editor's note: accepted, scheduled to Python 3.13), this PEP would supersede PEP 563, and PEP 563’s behavior would be deprecated and eventually removed.

In a nutshell, they are re-implementing __annotations__ to always return a (name -> type hint) dictionary, but changing its evaluation time from "always on module execution" to "by demand" by using a cached data-descriptorr.

george-zubrienko commented 11 months ago

@USSX-Hares then what would you recommend we do/not do about this?

george-zubrienko commented 10 months ago

Closing as wontfix, for now - README mentions that import should be avoided. Will revisit in v1.