Support for PEP 695

[erictraut]

PEP 695 was accepted, and it looks like it will make it into Python 3.12. This PEP adds support for a new type parameter syntax for generic classes and methods. It also adds a new syntax for type aliases (both generic and not).

PEP 695 functionality is implemented in pyright today. It would be great to see it also implemented in mypy and the other major type checkers so we could start to use the new features in type stubs.

Here is a rough task list associated with this work:

• [ ] Add support for infer_variance keyword parameter in TypeVar; see this section of PEP for details
• [x] Add support for variance inference (similar to variance enforcement in Protocol classes today); see this section of PEP for details
• [x] Add support for new AST nodes for type statement; see this section of PEP for details
• [x] Add support for new AST nodes for generic classes and functions; see this section of PEP for details
• [x] Add support for evaluating type parameters using the new syntax (including TypeVar bounds and constraints, TypeVarTuple and ParamSpec); see this section of PEP for details
• [x] Add support for defining type aliases using the new type statement (including generic and recursive type definitions); see this section of PEP for details
• [x] Add support for defining generic classes and functions with explicit type parameter lists using new syntax
• [x] Add support for functional form of TypeAliasType, supported for backward compatibility; see this section of PEP for details
• [x] Add detection and error reporting for use of traditional TypeVar, TypeVarTuple, or ParamSpec in conjunction with new syntax; see this section of PEP for details
• [ ] Add detection and error reporting for use of assignment expressions, yield, yield from, and await expressions within the type parameter scope; see this section of PEP for details

[AlexWaygood]

It would be great to see it also implemented in mypy and the other major type checkers so we could start to use the new features in type stubs.

Just to note — even if mypy implemented support for PEP 695 tomorrow, it will probably be quite a while before typeshed is able to use PEP 695 syntax, and I imagine the same will be true for many stub authors.

Mypy uses the Python ast module to parse stubs, meaning PEP 695 syntax will fail to be parsed by mypy when mypy is run on Python <=3.11. As such, we probably won't be able to adopt PEP 695 syntax in typeshed until we drop support for Python 3.11, unfortunately.

[abrahammurciano]

Python3.12 has been out for a while now, is there a timeline for when the new generics will be supported?

[JelleZijlstra]

When you (or anyone else reading this) submit a PR implementing it. Mypy is a volunteer-run project and has no set timelines.

I did a little preliminary work on this myself, but not sure I'll be able to complete it any time soon.

[berzi]

Haven't contributed to mypy yet, so, just asking: is it required that all the features of PEP 695 be implemented in the scope of one issue/PR? Is there something speaking against e.g. implementing support for the type keyword and merging it even if the other features aren't ready yet? Would we need to open separate issues for each in that case?

[JelleZijlstra]

No, implementing parts of it one by one is fine. We could consider using the --enable-incomplete-feature flag, like we did for Unpack, until everything is implemented.

Here's my partial attempt to implement support for type: https://github.com/python/mypy/commit/fd064fd6680b8b462c5440252cb00e1cbd0f4d62. Anyone who is interested, feel free to build off of that commit, or start from scratch as you prefer.

One thing I wasn't sure about was whether to create a new Node class for type statements (which is what I did in the commit), or try to fit it into the existing AssignmentNode. The former feels more principled and makes it easier to support type aliases with explicit type parameters; the second may lead to less code churn.

[Nachtalb]

No, implementing parts of it one by one is fine. We could consider using the --enable-incomplete-feature flag, like we did for Unpack, until everything is implemented.

That would be very appreciated 😄! As I can't seem to suppress this [valid-type] error :/

def chunks[T](seq: Sequence[T], n: int) -> Generator[Sequence[T], None, None]:  # type: ignore
    for i in range(0, len(seq), n):
        yield seq[i : i + n]

foobar/utils.py:8: error: PEP 695 generics are not yet supported
[valid-type]
    def chunks[T](seq: Sequence[T], n: int) -> Generator[Sequ...
               ^
Found 1 error in 1 file (checked 1 source file)

[bmitc]

How do I disable this check in a configuration file? It seems like it would make more sense to warn on this or ignore it rather than error if it's simply not supported by MyPy, but for now, how do I tell MyPy to globally not worry about it? Thank you!

Is

disable_error_code = valid-type

enough? Is valid-type specific to PEP 695 errors?

[bmitc]

I filed issue https://github.com/python/mypy/issues/16607 since it doesn't not appear possible to instruct Mypy to ignore this error via any of the existing ignore error methods. This seems somewhat critical to me in that it does not allow Mypy to be used in 3.12 projects requiring strict Mypy passes.

[JukkaL]

I started looking into this. So far the only tricky bit I've encountered is inferring variance. To infer variance, we need to know the types of all attributes, so it needs to happen after type checking -- but we need to know the variances of type variables to perform type checking.

The way we normally solve issues like this is to use deferral. Any code that needs to know the variance of a type variable defined in the same SCC would be deferred. After the first pass of type checking, and before type checking deferred targets again, we'd attempt to infer variances. The inference itself seems easy and very similar to what we already do when validating that variance is correct in protocols. We might need to do a few passes of variance inference in complex cases, and if inference fails, it may be best to assume the type variable is invariant (or covariant?) instead of generating an error.

@ilevkivskyi pointed out that we may need to defer module top levels, but this isn't supported right now. So it looks like we'll need to solve that first. Deferring top levels is very tricky when not using local partial types (--local-partial-types), so perhaps we'll first make local partial types enabled by default. We could only support the new generic syntax if using local partial types, but that seems acceptable. We've been planning to turn local partial types on for a long time, but this will cause a fairly big fallout, so we've postponed it until now. This will require increasing the major version of mypy as well. (We want local partial types enabled anyway, since mypy daemon requires this.)

Another idea I got from @ilevkivskyi is to build a tool to help migrate code to --local-partial-types. Building such a tool doesn't seem terribly difficult, and it would reduce the friction caused by the switch.

[JukkaL]

Many PEP 695 features are now supported on master. I'm hoping that we can have a useful implementation without deferring top levels or making other more complex changes, but it's not yet clear whether this will work out.

[JukkaL]

Note that PEP 695 support is currently behind this feature flags, since it's not complete yet: --enable-incomplete-feature=NewGenericSyntax

[JelleZijlstra]

Note for tracking: we now have experimental support for the new syntax, thanks to @JukkaL's work. I think we can close this issue once the support becomes non-experimental.

I added https://github.com/python/mypy/labels/topic-pep-695 to track any remaining or new issues.

[dbzix]

Note that PEP 695 support is currently behind this feature flags, since it's not complete yet: --enable-incomplete-feature=NewGenericSyntax

$ mypy  --enable-incomplete-feature=NewGenericSyntax .
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Unknown incomplete feature: NewGenericSyntax
$ mypy -V
mypy 1.10.0 (compiled: yes)

Am I missing something?

[AlexWaygood]

Note that PEP 695 support is currently behind this feature flags, since it's not complete yet: --enable-incomplete-feature=NewGenericSyntax

$ mypy  --enable-incomplete-feature=NewGenericSyntax .
usage: mypy [-h] [-v] [-V] [more options; see below]
            [-m MODULE] [-p PACKAGE] [-c PROGRAM_TEXT] [files ...]
mypy: error: Unknown incomplete feature: NewGenericSyntax
$ mypy -V
mypy 1.10.0 (compiled: yes)

Am I missing something?

Even the experimental support only exists on the master branch; it will be released as part of mypy 1.11.0.

[jstasiak]

For what it's worth I used this feature in a work project a week ago, running a version of Mypy straight from the master branch, and it worked flawlessly.

[lijok]

Works beautifully, well done team !