Support PEP 681 (dataclass_transform)

[NeilGirdhar]

PEP 681 introduces a new decorator function in the typing module named dataclass_transform. This was added to Python 3.11.

See #5383 for a related issue that this would solve. See #12840 for MyPy's Python 3.11 issues.

[NeilGirdhar]

It would be really nice to support frozen_default, which is not part of the PEP, but was added to CPython (https://github.com/python/cpython/issues/99957) and typing_extensions (https://github.com/python/typing_extensions/pull/101).

[tmke8]

Python 3.11 (to be released in one year)

Hm? Python 3.11 has already been released.

[NeilGirdhar]

@thomkeh Thanks for catching my mistake, edited 😄

[hauntsaninja]

Just to keep this issue up to date:

mypy 1.0 shipped with extremely rudimentary support for dataclass_transform decorator with no parameters passed. mypy master has increasingly better support for PEP 681.

[NeilGirdhar]

@wesleywright Looks like this complete on master? If so, I'll close this issue. Please feel free to tick this off the Python 3.11 issue.

[wesleywright]

@NeilGirdhar Yes, as far as I'm aware we're feature complete on master now.

[tmke8]

Does the implementation support taking into account overloads to determine whether a field will be present in the __init__? The master version in mypy play doesn't seem to support it: https://mypy-play.net/?mypy=master&python=3.11&gist=eec5f42639277ba0988d496cee03d3d4

The example is from this section of the PEP: https://peps.python.org/pep-0681/#field-specifier-parameters

[AlexWaygood]

Does the implementation support taking into account overloads to determine whether a field will be present in the __init__? The master version in mypy play doesn't seem to support it: https://mypy-play.net/?mypy=master&python=3.11&gist=eec5f42639277ba0988d496cee03d3d4

The "mypy master branch" option on mypy playground appears to be out of date with mypy's actual master branch (it often is).

If I have this snippet (slightly modified from your snippet above @tmke8):

import typing
from typing import overload, Optional, Any, Callable, Literal, Type, TypeVar
# Library code (within type stub or inline)
# In this library, passing a resolver means that init must be False,
# and the overload with Literal[False] enforces that.
@overload
def model_field(
        *,
        default: Optional[Any] = ...,
        resolver: Callable[[], Any],
        init: Literal[False] = False,
    ) -> Any: ...

@overload
def model_field(
        *,
        default: Optional[Any] = ...,
        resolver: None = None,
        init: bool,
    ) -> Any: ...
def model_field(
    *,
    default: Optional[Any] = ...,
    resolver: Optional[Callable[[], Any]] = ...,
    init: bool = ...,
) -> Any: ...

_T = TypeVar("_T")
@typing.dataclass_transform(
    kw_only_default=True,
    field_specifiers=(model_field, ))
def create_model(
    *,
    init: bool = True,
) -> Callable[[Type[_T]], Type[_T]]: ...

# Code that imports this library:
@create_model(init=True)
class CustomerModel:
    id: int = model_field(resolver=lambda : 0)
    name: str

cm = CustomerModel("John")

Here's mypy's output using its actual master branch:

test.py:32: error: Missing return statement  [empty-body]
test.py:43: error: Too many positional arguments for "CustomerModel"  [misc]
test.py:43: error: Missing named argument "name" for "CustomerModel"  [call-arg]
test.py:43: error: Argument 1 to "CustomerModel" has incompatible type "str"; expected "int"  [arg-type]

Is anything there unexpected?

[AlexWaygood]

Mypy-playground is a third-party project that's not maintained by mypy devs; here are two open issues about the "mypy master branch" option being quite confusing and often out-of-date:

• https://github.com/ymyzk/mypy-playground/issues/158
• https://github.com/ymyzk/mypy-playground/issues/162

[AlexWaygood]

Re-reading the PEP, it looks like the use of overloads for field specifiers isn't quite fully implemented. We currently have this behaviour on mypy master:

import typing
from typing import overload, Optional, Any, Callable, Literal, Type, TypeVar
# Library code (within type stub or inline)
# In this library, passing a resolver means that init must be False,
# and the overload with Literal[False] enforces that.
@overload
def model_field(
        *,
        default: Optional[Any] = ...,
        resolver: Callable[[], Any],
        init: Literal[False] = False,
    ) -> Any: ...

@overload
def model_field(
        *,
        default: Optional[Any] = ...,
        resolver: None = None,
        init: bool = True,
    ) -> Any: ...
def model_field(
    *,
    default: Optional[Any] = ...,
    resolver: Optional[Callable[[], Any]] = ...,
    init: bool = ...,
) -> Any: ...

_T = TypeVar("_T")
@typing.dataclass_transform(
    kw_only_default=True,
    field_specifiers=(model_field, ))
def create_model(
    *,
    init: bool = True,
) -> Callable[[Type[_T]], Type[_T]]: ...

# Code that imports this library:
@create_model(init=True)
class CustomerModel:
    id: int = model_field(resolver=lambda : 0)
    name: str

cm = CustomerModel(name="John")

Mypy output:

test.py:32: error: Missing return statement  [empty-body]
test.py:43: error: Missing named argument "id" for "CustomerModel"  [call-arg]
Found 2 errors in 1 file (checked 1 source file)

The second error is a false positive: mypy should be able to infer that there's no "id" argument in the __init__ method for CustomerModel, due to the fact that the resolver argument was specified for model_field for the id attribute.

[wesleywright]

Opened https://github.com/python/mypy/pull/14870 to support the implicit defaults for the init parameter, though I'm not sure how to plumb the correct overload in, so it only works for non-overloads for now. Hopefully that should be easy to fix and the feature will be properly supported.

[wesleywright]

@tmke8 are you aware of any specific use cases for such overloads in the wild?

[tmke8]

@wesleywright no, I just read about them in the PEP

[AlexWaygood]

@tmke8 are you aware of any specific use cases for such overloads in the wild?

You could check with the pydantic and SQLAlchemy folks; they seem to be heavy users of dataclass_transform

[hauntsaninja]

You can see more context about the field specifier overloads in https://github.com/microsoft/pyright/discussions/1782?sort=old#discussioncomment-1268813 cc @patrick91

[tmke8]

More specific link: https://github.com/microsoft/pyright/discussions/1782#discussioncomment-1229854

[JukkaL]

I suspect that full overload resolution in the dataclass plugin would require some changes to the plugin system, since during the main dataclass transform pass types aren't fully set up yet, and subtype checks can't be reliably used. I wonder if it would be sufficient to do "lightweight" overload resolution using only the argument counts and names and some simple type rules (e.g. matching None value to None type). This might unblock the currently known use cases while we figure out how to do this in a more general way.

A possible more general approach would to postpone the determination of the __init__ signature until just before type checking, when all types are fully set up. Even then it may be tricky to use the existing machinery to match overloads.

Yet another idea would be to generate the __init__ signature lazily during type checking. I think this could also hit some difficulties, but I'm not sure.

[NeilGirdhar]

I believe this is complete now.