Investigate support for recursive Union types

[JWCS]

• mashumaro version: 3.12
• Python version: 3.12.2
• Operating System: alpine 3.19 docker

Description

In the past, I've used the below JSON type definitions, which have worked well with both static and runtime type checkers (mypy (except the new one, wip) and beartype). I had a message type that included an arbitrary json blob as a field. Attempting to serialize that with mashumaro led to an error. I'm not entirely sure if this recursive functionality is supported by mashumaro, due to pre-compilation, only that the underlying types themselves are. If there's a better workaround, I apologies if I missed the documentation.

What I Did

I tried to use both the "new" 3.12 type keyword definition of JSON, and the "old" recursive style (avoiding from __future__ import annotations).

from mashumaro.mixins.orjson import DataClassORJSONMixin
from dataclasses import dataclass
from typing import Union
type JSON = Union[dict[str, JSON], list[JSON], str, int, float, bool, None]
@dataclass
class MsgT(DataClassORJSONMixin):
    name: str
    meta: str
    msg: JSON

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.12/site-packages/mashumaro/mixins/dict.py", line 24, in __init_subclass__
    compile_mixin_unpacker(cls, **builder_params["unpacker"])
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/mixin.py", line 49, in compile_mixin_unpacker
    builder.add_unpack_method()
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 557, in add_unpack_method
    self._add_unpack_method_lines(method_name)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 462, in _add_unpack_method_lines
    ).build(
      ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 1276, in build
    unpacked_value = UnpackerRegistry.get(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 806, in unpack_special_typing_primitive
    raise UnserializableDataError(
mashumaro.exceptions.UnserializableDataError: JSON as a field type is not supported by mashumaro

from mashumaro.mixins.orjson import DataClassORJSONMixin
from dataclasses import dataclass
from typing import Union
JSON = Union[dict[str, 'JSON'], list['JSON'], str, int, float, bool, None]
@dataclass
class MsgT(DataClassORJSONMixin):
    name: str
    meta: str
    msg: JSON

Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
  File "/usr/local/lib/python3.12/site-packages/mashumaro/mixins/dict.py", line 24, in __init_subclass__
    compile_mixin_unpacker(cls, **builder_params["unpacker"])
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/mixin.py", line 49, in compile_mixin_unpacker
    builder.add_unpack_method()
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 557, in add_unpack_method
    self._add_unpack_method_lines(method_name)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 462, in _add_unpack_method_lines
    ).build(
      ^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 1276, in build
    unpacked_value = UnpackerRegistry.get(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 720, in unpack_special_typing_primitive
    return UnionUnpackerBuilder(union_args).build(spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 212, in build
    self._add_body(spec, lines)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 169, in _add_body
    for unpacker in (
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 170, in <genexpr>
    UnpackerRegistry.get(spec.copy(type=type_arg, expression="value"))
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1219, in unpack_collection
    f'{{{inner_expr(0, "key")}: {inner_expr(1)} '
                                 ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1153, in inner_expr
    return UnpackerRegistry.get(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 720, in unpack_special_typing_primitive
    return UnionUnpackerBuilder(union_args).build(spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 212, in build
    self._add_body(spec, lines)
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 169, in _add_body
    for unpacker in (
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 170, in <genexpr>
    UnpackerRegistry.get(spec.copy(type=type_arg, expression="value"))
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1219, in unpack_collection
    f'{{{inner_expr(0, "key")}: {inner_expr(1)} '
                                 ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 1153, in inner_expr
    return UnpackerRegistry.get(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/common.py", line 238, in get
    expr = packer(spec)
           ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/types/unpack.py", line 801, in unpack_special_typing_primitive
    evaluated = spec.builder.evaluate_forward_ref(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/code/builder.py", line 339, in evaluate_forward_ref
    return evaluate_forward_ref(typ, globalns, self.__dict__)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/mashumaro/core/meta/helpers.py", line 769, in evaluate_forward_ref
    return typ._evaluate(
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/typing.py", line 907, in _evaluate
    eval(self.__forward_code__, globalns, localns),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'JSON' is not defined

[Fatal1ty]

Hi @JWCS

I tried to use both the "new" 3.12 type keyword definition of JSON

The type statement is not yet supported, but this is a good improvement that would be good to include in the next release. It should be trivial.

and the "old" recursive style (avoiding from future import annotations).

Recursion is a problem here. I have not encountered such a case yet, so it has not been tested. It will take more time to figure out if this case can be supported. I'm curious in which real case such a broad recursive type is needed. Could you tell me in more detail about how you are going to use this JSON type?

[JWCS]

Thanks for the response. I actually wasn't sure if I was being oblivious in the documentation, and missing this recursive case. I know there's support for custom types... but compared to the lazy solution (below), I was hoping it was me. For recursion, to be honest, I'm pretty heavy with typing, but the only "recursive" definition I've ever seen (of use) is this JSON one, or something that looks like it (read only, int keys, same enough). A solution in the general area of support is likely sufficient (see below).

My real world use case is for dealing with abstract customer json payloads, and validating no (structural) corruption in transmission. For example, given a bunch of header fields specifying the payload conditions, the payload itself is json; I don't care what the json is, but it is in that format, and I would like to make sure there was no corruption in transmission. For example, I played with the idea of not serializing the payload, msg: bytes, leaving it as a bytes stream, not validating the payload's structural integrity, but that's not the actual encoding.

Alternatively, what I am currently doing (the show must move on), is just msg: dict[str, Any], which mashumaro happily accepts. And in practice this might be sufficient enough to just be the documented answer. "Just use Any". The only downside is that the type then isn't JSON, so there's duplicated type hints. In terms of typechecking, dict[str, Any] > dict[str, JSON], which leads to some complaints. But that's not critical; runtime typecheckers test against what's there, which is json, and mypy can be coerced.