lidatong / dataclasses-json

Easily serialize Data Classes to and from JSON
MIT License
1.34k stars 150 forks source link

[BUG] 0.6.2 changes CatchAll behavior in unexpected ways #499

Closed jasonrock-a3 closed 7 months ago

jasonrock-a3 commented 7 months ago

Description

The changes in 0.6.2 appear to introduce some new behavior when dataclasses with CatchAll are inherited from. A workaround is to include the CatchAll field in all subclasses, but that seems to break away from how other fields work.

Code snippet that reproduces the issue

from dataclasses import dataclass, field
from dataclasses_json import CatchAll, Undefined, dataclass_json

@dataclass_json(undefined=Undefined.INCLUDE)
@dataclass
class TestInternalConfig:
    options: CatchAll = None
    val: str = "bar"

@dataclass_json(undefined=Undefined.INCLUDE)
@dataclass
class TestInternalExtendConfig(TestInternalConfig):
    val: str = "baz"

conf = TestInternalExtendConfig()
conf.to_dict()
Traceback (most recent call last):
  File "test.py", line 47, in <module>
    conf.to_dict()
  File "lib/python3.9/site-packages/dataclasses_json/api.py", line 73, in to_dict
    return _asdict(self, encode_json=encode_json)
  File "lib/python3.9/site-packages/dataclasses_json/core.py", line 414, in _asdict
    value = _asdict(
  File "lib/python3.9/site-packages/dataclasses_json/core.py", line 420, in _asdict
    result = _handle_undefined_parameters_safe(cls=obj, kvs=dict(result),
  File "python3.9/site-packages/dataclasses_json/utils.py", line 200, in _handle_undefined_parameters_safe
    return undefined_parameter_action.value.handle_to_dict(obj=cls,
  File "lib/python3.9/site-packages/dataclasses_json/undefined.py", line 201, in handle_to_dict
    _CatchAllUndefinedParameters._get_catch_all_field(obj)
  File "lib/python3.9/site-packages/dataclasses_json/undefined.py", line 252, in _get_catch_all_field
    catch_all_fields = list(
  File "lib/python3.9/site-packages/dataclasses_json/undefined.py", line 253, in <lambda>
    filter(lambda f: types[f.name] == Optional[CatchAllVar], fields(cls)))
KeyError: 'options'

Describe the results you expected

In previous versions this wouldn't error, as expected, since options is defined in the superclass TestInternalConfig

Python version you are using

Python 3.9.18

Environment description

python -m pip freeze dataclasses-json==0.6.2 marshmallow==3.20.1 mypy-extensions==1.0.0 packaging==23.2 typing-inspect==0.9.0 typing_extensions==4.8.0

jasonrock-a3 commented 7 months ago

Slight updates: no need to do the TestConfig.

Also I wasn't quite right about the impact. It looks like in order to extend a Dataclass with catchall, you now have to define all of the fields in the subclass.

from dataclasses import dataclass, field
from dataclasses_json import CatchAll, Undefined, dataclass_json

@dataclass_json(undefined=Undefined.INCLUDE)
@dataclass
class TestInternalConfig:
    options: CatchAll = None
    val: str = "bar"
    val2: int = 0 

@dataclass_json(undefined=Undefined.INCLUDE)
@dataclass
class TestInternalExtendConfig1(TestInternalConfig):
    options: CatchAll = None
    val: str = "baz"
    val2: int = 0 

@dataclass_json(undefined=Undefined.INCLUDE)
@dataclass
class TestInternalExtendConfig2(TestInternalConfig):
    options: CatchAll = None
    val: str = "baz"

# This will succeed
tie1 = TestInternalExtendConfig1()
tie1.to_dict()

# This will fail
tie2 = TestInternalExtendConfig2()
tie2.to_dict()
Traceback (most recent call last):
  File "test.py", line 51, in <module>
    tie2.to_dict()
  File "python3.9/site-packages/dataclasses_json/api.py", line 73, in to_dict
    return _asdict(self, encode_json=encode_json)
  File "python3.9/site-packages/dataclasses_json/core.py", line 420, in _asdict
    result = _handle_undefined_parameters_safe(cls=obj, kvs=dict(result),
  File "python3.9/site-packages/dataclasses_json/utils.py", line 200, in _handle_undefined_parameters_safe
    return undefined_parameter_action.value.handle_to_dict(obj=cls,
  File "python3.9/site-packages/dataclasses_json/undefined.py", line 201, in handle_to_dict
    _CatchAllUndefinedParameters._get_catch_all_field(obj)
  File "python3.9/site-packages/dataclasses_json/undefined.py", line 252, in _get_catch_all_field
    catch_all_fields = list(
  File "python3.9/site-packages/dataclasses_json/undefined.py", line 253, in <lambda>
    filter(lambda f: types[f.name] == Optional[CatchAllVar], fields(cls)))
KeyError: 'val2'

I traced the issue some, and it seems like what's happening is that types = get_type_hints(cls, globalns=cls_globals) is called three times, twice during initialization which work, and once during to_dict which fails.

  File "test.py", line 50, in <module>
    tie2 = TestInternalExtendConfig2()
  File "dataclasses-json/dataclasses_json/undefined.py", line 226, in _catch_all_init
    if _CatchAllUndefinedParameters._get_catch_all_field(
  File "dataclasses-json/dataclasses_json/undefined.py", line 251, in _get_catch_all_field
    traceback.print_stack()

  File "test.py", line 50, in <module>
    tie2 = TestInternalExtendConfig2()
  File "dataclasses-json/dataclasses_json/undefined.py", line 242, in _catch_all_init
    final_parameters = _CatchAllUndefinedParameters.handle_from_dict(
  File "dataclasses-json/dataclasses_json/undefined.py", line 138, in handle_from_dict
    catch_all_field = _CatchAllUndefinedParameters._get_catch_all_field(
  File "dataclasses-json/dataclasses_json/undefined.py", line 251, in _get_catch_all_field
    traceback.print_stack()

  File "test.py", line 51, in <module>
    tie2.to_dict()
  File "dataclasses-json/dataclasses_json/api.py", line 73, in to_dict
    return _asdict(self, encode_json=encode_json)
  File "dataclasses-json/dataclasses_json/core.py", line 420, in _asdict
    result = _handle_undefined_parameters_safe(cls=obj, kvs=dict(result),
  File "dataclasses-json/dataclasses_json/utils.py", line 200, in _handle_undefined_parameters_safe
    return undefined_parameter_action.value.handle_to_dict(obj=cls,
  File "dataclasses-json/dataclasses_json/undefined.py", line 201, in handle_to_dict
    _CatchAllUndefinedParameters._get_catch_all_field(obj)
  File "dataclasses-json/dataclasses_json/undefined.py", line 251, in _get_catch_all_field
    traceback.print_stack()
george-zubrienko commented 7 months ago

Linked fix released in v0.6.3 @jasonrock-a3