Support recursive types

[o11c]

The following in particular would be useful:

Callback = Callable[[str], 'Callback']
Foo = Union[str, List['Foo']]

[Mattwmaster58]

This is the most thumbed-up open issue by a large margin

[ethanhs]

@Mattwmaster58 unfortunately it is also rather complicated, as it requires a lot of changes to mypy. Ivan has done a lot of work on this, but there is still more to do, and he isn't working on it anymore.

We understand there is great interest and need for recursive types, but open source works at the speed of people's spare time and interest.

[LivInTheLookingGlass]

Is there anything that newcomers to the project might be able to realistically help with this?

[Dreamsorcerer]

For anyone else trying to do recursive Callables. A callback protocol seems to work fine:

class Foo(Protocol):
    def __call__(self) -> Awaitable['Foo']: ...

or similar.

[craigh92]

I would also like to express interest in this feature. In particular:

NestedStrDict = Dict[str, Union[NestedStrDict, int]]

For objects such as:

obj = {
    'a' : 0,
    'nested_dict' : 
    { 
        'b' : 1,
        'c' : 2,
        'nested2_dict' : 
        {
             'd' : 3
        }
    }
}

[sg495]

Edit: I did not see the comment https://github.com/python/mypy/issues/731#issuecomment-423845865 by @jhrmnn when first reading through this issue. I guess I am proposing the same thing and wondering exactly how much work remains to be done before we get a protocol-based solution to (some) recursive types, like the one described in that/this comment. (I should disclaim that I am an avid user of Mypy but not sufficiently acquainted with the nitty-gritty of its implementation. That said: if it is realistic for me to help with this issue, I'm certainly happy to help.)

Original Comment

It seems that some additional support for list literals and dictionary literals might make it possible to implement a recursive JSON-like type using protocols, as follows:

from typing import Iterator, Protocol, runtime_checkable, Union
from typing import KeysView, ValuesView, ItemsView

@runtime_checkable
class JSONSequence(Protocol):
    """
        Protocol for a sequence of JSON values.
    """

    def __getitem__(self, idx: int) -> "JSONLike":
        ...

    def __contains__(self, value: "JSONLike") -> bool:
        ...

    def __len__(self) -> int:
        ...

    def __iter__(self) -> Iterator["JSONLike"]:
        ...

    def __reversed__(self) -> Iterator["JSONLike"]:
        ...

@runtime_checkable
class JSONMapping(Protocol):
    """
        Protocol for a mapping of strings to JSON values.
    """

    def __getitem__(self, key: str) -> "JSONLike":
        ...

    def __contains__(self, key: str) -> bool:
        ...

    def __len__(self) -> int:
        ...

    def __iter__(self) -> Iterator[str]:
        ...

    def keys(self) -> KeysView[str]:
        ...

    def values(self) -> ValuesView["JSONLike"]:
        ...

    def items(self) -> ItemsView[str, "JSONLike"]:
        ...

JSONLike = Union[None, bool, int, str, JSONSequence, JSONMapping]
""" Recursive JSON type. """

Indeed, it seems that Mypy (v 0.782) already recognises as correct some of the interesting combinations, and only requires little nudging in other cases.

# ... Continued from before ...
from typing import cast

x: JSONLike = None                                  # OK
x = True                                            # OK
x = 42                                              # OK
x = "hello"                                         # OK
x = []                                              # OK
x = [None, True, 43, "hello"]                       # OK
x = {"the final answer": 42}                        # OK
x = {"burn baby burn": "master ignition routine"}   # OK
x = {
    "the final answer": cast(JSONLike, 42),
    "burn baby burn": "master ignition routine"
}                                                   # OK
x = {
    "the final answer": cast(JSONLike, 42),
    "burn baby burn": "master ignition routine",
    "a bunch of stuff": {
        "none": None,
        "bool": True,
        "int": 43,
        "str": "hello",
        "list": [None, True, 43, "hello"]
    }
}                                                   # OK

x = {}
# Mypy Error: assignment - Incompatible types in assignment
# expression has type "Dict[<nothing>, <nothing>]",
# variable has type "Union[None, bool, int, str, JSONSequence, JSONMapping]"

x = {
    "the final answer": 42,
    "burn baby burn": "master ignition routine"
}
# Mypy Error: assignment - Incompatible types in assignment
# expression has type "Dict[str, object]",
# variable has type "Union[None, bool, int, str, JSONSequence, JSONMapping]"

One has to be a little careful with runtime typechecking, e.g. because strings and dictionaries match the protocol for JSON sequences (at least as written above), but I'm confident that this could be remedied with little additional work:

# ... Continued from before ...
from collections import deque
from typing import Dict

def to_strict_json(json_like: JSONLike) -> Union[None, bool, int, str, list, dict]:
    """
        Recursively converts `JSONSequence`s to `list`s
        and `JSONMapping`s to `dict`s.
    """
    if isinstance(json_like, str):
        # strings match the `JSONSequence` protocol, must intercept.
        return json_like
    if isinstance(json_like, JSONMapping):
        # dictionaries match the `JSONSequence` protocol, must intercept
        return {k: to_strict_json(v) for k, v in json_like.items()}
    if isinstance(json_like, JSONSequence):
        return [to_strict_json(x) for x in json_like]
    return json_like

class MyReadonlyDict(JSONMapping):
    """ A silly readonly dictionary wrapper. """
    _dict: Dict[str, JSONLike]
    def __init__(self, d):
        super().__init__()
        self._dict = dict(d)
    def __getitem__(self, key: str) -> JSONLike:
        return self._dict[key]
    def __contains__(self, key: str) -> bool:
        return key in self._dict
    def __len__(self) -> int:
        return len(self._dict)
    def __iter__(self) -> Iterator[str]:
        return iter(self._dict)
    def keys(self) -> KeysView[str]:
        return self._dict.keys()
    def values(self) -> ValuesView[JSONLike]:
        return self._dict.values()
    def items(self) -> ItemsView[str, JSONLike]:
        return self._dict.items()

x = deque([None, True, 43, MyReadonlyDict({'a str': ['hello', 'bye bye']})]) # OK
print(to_strict_json(x))
# [None, True, 43, {'a str': ['hello', 'bye bye']}]

PS: I'm just throwing this out there as an instance of protocols already being "close enough" for some recursive types to be written. I'm not saying that this would necessarily be a full solution for a JSON type (cf https://github.com/python/typing/issues/182).

[altvod]

If I remember correctly, # type: ignore used to suppress recursive type errors, but in the current version (mypy==0.790) it doesn't seem to work - I'm seeing a lot of these errors in spite of # type: ignore. Am I missing something?

[antonagestam]

@altvod https://github.com/python/mypy/issues/7069

[altvod]

@antonagestam that issue is closed. Does this mean that there isn't (and will not be) a way to suppress these errors until recursive types are fully implemented?

[antonagestam]

@altvod I think that is a safe assumption to make.

[intgr]

An option to suppress errors due to recursive types is a very reasonable request, and far easier to implement than full support for recursive types. Especially now that this issue is basically disowned, it's not clear when someone will pick it up again.

Please open a separate issue for the suppresion capability.

[altvod]

I realized that I had in fact a slightly different error: #8695

[csenn]

It's clear this issue is complicated, but just want to provide a TypedDict recursive example that is probably pretty common and failing for anyone searching.

Self Reference:

from typing import TypedDict, List

Comment = TypedDict('Comment', {
  "text": str,
  "childComments": List['Comment']
})

Deep Reference:

from typing import TypedDict, List

Comment = TypedDict('Comment', {
  "text": str,
  "reply": "CommentThread"
})

CommentThread = TypedDict('CommentThread', {
  "likes": int,
  "comments": List[Comment]
})

Both display error: possible cyclic definition

[hmc-cs-mdrissi]

JSON = Union[Dict[str, "JSON"], List["JSON"], str, int, float, bool, None] This example while classic case plays poorly with the type variance rules in python. You would hope that List[int] is a JSON type but it is not due to variance. Expanding the JSON type for the list case it is,

List[Union[Dict[str, "JSON"], List["JSON"], str, int, float, bool, None]]]

List[int] is not a subtype of List[Union[int, ...]]. You can make covariant json if you are willing to use mapping/sequence but if you want an actual list/dict or intend your json type to be mutable it'll be hard to keep the type variance rules and have useful recursive types involving any invariant generic.

pyright has recursive type support, but actually writing a json type with list/dict and using list with it don't cooperate with the variance rules. I think any solution would require updating the variance rules specifically for recursive types.

edit: My understanding was wrong thinking/discussing it more. The issue goes away if you are fine with list/dict elements being non homogenous (so json).

[BvB93]

Ever since the most recent mypy release there seems to be some basic support for recursive types, as can been seen in the recursive protocol version of Sequence below:

from __future__ import annotations

from collections.abc import Iterator
from typing import TypeVar, Protocol, overload, Any, TYPE_CHECKING

_T_co = TypeVar("_T_co")

class _RecursiveSequence(Protocol[_T_co]):
    def __len__(self) -> int: ...
    @overload
    def __getitem__(self, __index: int) -> _T_co | _RecursiveSequence[_T_co]: ...
    @overload
    def __getitem__(self, __index: slice) -> _RecursiveSequence[_T_co]: ...
    def __contains__(self, __x: object) -> bool: ...
    def __iter__(self) -> Iterator[_T_co | _RecursiveSequence[_T_co]]: ...
    def __reversed__(self) -> Iterator[_T_co | _RecursiveSequence[_T_co]]: ...
    def count(self, __value: Any) -> int: ...
    def index(self, __value: Any, __start: int = ..., __stop: int = ...) -> int: ...

def func1(a: _RecursiveSequence[int]) -> int: ...

if TYPE_CHECKING:
    reveal_type(func1([1]))         # Revealed type is "builtins.int"
    reveal_type(func1([[1]]))       # Revealed type is "builtins.int"
    reveal_type(func1([[[1]]]))     # Revealed type is "builtins.int"
    reveal_type(func1((1, 2, 3)))   # Revealed type is "builtins.int"
    reveal_type(func1([(1, 2, 3)])) # Revealed type is "builtins.int"
    reveal_type(func1([True]))      # Revealed type is "builtins.int"

The only area where mypy still fails is if typevars are involved. This is a bit of a shame, but the fact that there is now basic support for recursive types is already a big step forward.

_T = TypeVar("_T")

def func2(a: npt._NestedSequence[_T]) -> _T: ...

seq_1d_a: list[int]
seq_1d_b = [1]
seq_2d_a: list[list[int]]
seq_2d_b = [[1]]

if TYPE_CHECKING:
    # The good
    reveal_type(func2(seq_1d_a))  # Revealed type is "builtins.int*"

    # The bad
    reveal_type(func2(seq_1d_b))  # Revealed type is "Any"
    reveal_type(func2(seq_2d_a))  # Argument 1 to "func" has incompatible type "List[List[int]]"; expected "_NestedSequence[<nothing>]"
    reveal_type(func2(seq_2d_b))  # Revealed type is "Any"

[charbonnierg]

Ever since the most recent mypy release there seems to be some basic support for recursive types, as can been seen in the recursive protocol version of Sequence below:

from __future__ import annotations

from collections.abc import Iterator
from typing import TypeVar, Protocol, overload, Any, TYPE_CHECKING

_T_co = TypeVar("_T_co")

class _RecursiveSequence(Protocol[_T_co]):
    def __len__(self) -> int: ...
    @overload
    def __getitem__(self, __index: int) -> _T_co | _RecursiveSequence[_T_co]: ...
    @overload
    def __getitem__(self, __index: slice) -> _RecursiveSequence[_T_co]: ...
    def __contains__(self, __x: object) -> bool: ...
    def __iter__(self) -> Iterator[_T_co | _RecursiveSequence[_T_co]]: ...
    def __reversed__(self) -> Iterator[_T_co | _RecursiveSequence[_T_co]]: ...
    def count(self, __value: Any) -> int: ...
    def index(self, __value: Any, __start: int = ..., __stop: int = ...) -> int: ...

def func1(a: _RecursiveSequence[int]) -> int: ...

if TYPE_CHECKING:
    reveal_type(func1([1]))         # Revealed type is "builtins.int"
    reveal_type(func1([[1]]))       # Revealed type is "builtins.int"
    reveal_type(func1([[[1]]]))     # Revealed type is "builtins.int"
    reveal_type(func1((1, 2, 3)))   # Revealed type is "builtins.int"
    reveal_type(func1([(1, 2, 3)])) # Revealed type is "builtins.int"
    reveal_type(func1([True]))      # Revealed type is "builtins.int"

The only area where mypy still fails is if typevars are involved. This is a bit of a shame, but the fact that there is now basic support for recursive types is already a big step forward.

_T = TypeVar("_T")

def func2(a: npt._NestedSequence[_T]) -> _T: ...

seq_1d_a: list[int]
seq_1d_b = [1]
seq_2d_a: list[list[int]]
seq_2d_b = [[1]]

if TYPE_CHECKING:
    # The good
    reveal_type(func2(seq_1d_a))  # Revealed type is "builtins.int*"

    # The bad
    reveal_type(func2(seq_1d_b))  # Revealed type is "Any"
    reveal_type(func2(seq_2d_a))  # Argument 1 to "func" has incompatible type "List[List[int]]"; expected "_NestedSequence[<nothing>]"
    reveal_type(func2(seq_2d_b))  # Revealed type is "Any"

I saw the module _nested_sequence.py available in numpy, so I tried to copy/paste it into my project, but I can't get it to work for simple use cases:

v: _NestedSequence[int]
v1: _NestedSequence[int]
v2: _NestedSequence[int]
v3: _NestedSequence[int]
v4: _NestedSequence[int]

# Fails as expected: 
# Incompatible types in assignment (expression has type "int", variable has type "_NestedSequence[int]")  [assignment]mypy(error)
v = 1

# Does not fail as expected
v1 = [1]
# Does not fail as expected
v2 = [[1]]

# Does not fail, but I expected a failure
v3 = ["a"]
v4 = [["a"]]

@BvB93 Are you aware of such behavior or is there a problem on my side ?

I'm using Python 3.8.10, with mypy==0.942 and mypy-extensions==0.4.3.

My mypy config is:

[mypy]
exclude = notebooks
scripts_are_modules = True
show_traceback = True
disallow_incomplete_defs = True
check_untyped_defs = True
disallow_untyped_defs = True
disallow_untyped_decorators = True
disallow_any_generics = True
warn_no_return = True
no_implicit_optional = True
strict_optional = True
warn_redundant_casts = True
warn_unused_ignores = True
warn_return_any = True
warn_unreachable = True
show_error_codes = True
show_column_numbers = True
ignore_missing_imports = True
implicit_reexport = True
plugins = pydantic.mypy

Thanks and happy easter !

EDIT: I created a reproducible example

[eugene-bright]

Pyright/Pylance has recursive type definition support. It's good idea to mimic that https://devblogs.microsoft.com/python/pylance-introduces-five-new-features-that-enable-type-magic-for-python-developers/

[charbonnierg]

I understand now that the problem only occurs when using literal values which are not annotated. Using _NestedSequence as implemented in numpy.typing:

def func1(a: _NestedSequence[int]) -> int: ...

# Fails as expected:
# Incompatible types in assignment (expression has type "int", variable has type "_NestedSequence[int]")  [assignment]mypy(error)
v1 = func1(1)   # type: ignore[arg-type]

# Does not fail as expected
v2 = func1([1])
# Does not fail as expected
v3 = func1([[1]])

# Does not fail
v4 = func1(["a"])  # [arg-type] error was expected
v5 = func1([["a"]])  # [arg-type] error was expected

# Does fail as expected
input_: List[str] = ["a"]
input__: List[List[str]] = [["a"]]
v4_ = func1(input_) # type: ignore[arg-type]
v5 = func1(input__) # type: ignore[arg-type]

[erictraut]

As @eugene-bright mentioned, pyright supports recursive type aliases, allowing for simple recursive type definitions like this:

_RecursiveSequence = _T_co | Sequence["_RecursiveSequence[_T_co]"]

This works with all of the examples above.

def func1(a: _RecursiveSequence[_T_co]) -> _T_co:
    ...

reveal_type(func1(1))  # Revealed type is "int"
reveal_type(func1([1]))  # Revealed type is "int"
reveal_type(func1([[1]]))  # Revealed type is "int"
reveal_type(func1([[[1]]]))  # Revealed type is "int"
reveal_type(func1((1, 2, 3)))  # Revealed type is "int"
reveal_type(func1([(1, 2, 3)]))  # Revealed type is "int"
reveal_type(func1([True]))  # Revealed type is "bool"

reveal_type(func1(["a"]))  # Revealed type is "str"
reveal_type(func1([["a"]]))  # Revealed type is "str"

input1: list[str] = ["a"]
input2: list[list[str]] = [["a"]]
reveal_type(func1(input1))  # Revealed type is "str"
reveal_type(func1(input2))  # Revealed type is "str"

Pyre supports recursive type aliases as well. It would be great if mypy added this support so library authors could make use of the feature.

[charbonnierg]

But why are you all talking about pyright? Unless I'm missing something, we're writing comments on the mypy repository ? Switching to pyright right now for the sole benefice of recursive typing is out of question for me at the moment. Moreover, I'm quite satisfied with the basic recursion features mypy offer. My question concerns a behavior of mypy which I found strange, (maybe incorrect?) encountered when trying a solution proposed in this thread. Maybe I should have open an issue on numpy repository, but I thought it was more of a mypy issue and did not want to create a duplicate issue...

[JelleZijlstra]

Eric is the author of pyright, so it's hardly surprising that he talks about it :) I think it's helpful to consider what other type checkers do as we consider how to add recursive type support to mypy.

[charbonnierg]

Sorry if my answer seemed harsh, I can only agree with your statement. I do realize that I was looking for an anwser to my question more than discussing how to implement recursive types, my bad. Do you know where such question should be asked though ?

[JelleZijlstra]

Maybe open a new topic at https://github.com/python/typing/discussions

[charbonnierg]

Thanks, it's done, feel free to remove my comments as they do not contribute much to the discussion.

[BvB93]

@charbonnierg I don’t have time to look at it right now, but I do recall seeing some oddities with strings before. My guess is that it’s string’s recursive definition (class str(Sequence[str]): …) that eventually causes mypy to cast it to _NestedSequence[Any] in the example you provided above. This is still a bit surprising though, as both types have incompatible __contains__ definitions, regardless of generic parameter.

[davidhalter]

Mypy actually supports recursive types in classes very well, it just does not support recursive type aliases:

from typing import *                                                            
T = TypeVar("T")                                                                

class X(list[X[T] | T]): ...                                                    

foo: X[int] = X([X([1])])                                                       
reveal_type(foo)                                                                
reveal_type(foo[0]) 

$ mypy new.py --strict
new.py:7: note: Revealed type is "new.X[builtins.int]"
new.py:8: note: Revealed type is "Union[new.X[builtins.int*], builtins.int*]"

Only posting this, because I haven't seen an example like this before. It might not be very useful, but it's still recursive :)

[ThatXliner]

Only posting this, because I haven't seen an example like this before. It might not be very useful, but it's still recursive :)

Possibly useful example:

Trie = Dict[str, Union["Trie", Literal["$END"]]]

Another realistic example: type annotating JSON's return type

[gvanrossum]

We don’t need more examples of why this is useful. We know. We just need someone to implement it. So I have locked the issue.

[ilevkivskyi]

I think this can now be closed by https://github.com/python/mypy/pull/13297 (and follow-up PRs). If there are remaining problems, they can be opened as separate issues.

Note this is only enabled in master with --enable-recursive-aliases (enables recursive type aliases, recursive TypedDicts, and recursive NamedTuples; as mentioned above proper classes and protocols can already be recursive). Note this is still considered an experimental feature, in particular type inference may sometimes fail, if this happens you can try switching to covariant collections, and/or simply give an explicit annotation.

[cj81499]

@ilevkivskyi is there a way to set this config option in a pyproject.toml file? I checked the documentation, but couldn't find anything about the new cli flag (perhaps because it's still experimental, and will eventually be enabled by default?)

[ilevkivskyi]

Yes, it is indeed a temporary flag (for few releases), and in 0.990 it will be on by default, you can try

$ cat pyproject.toml
[tool.mypy]
enable_recursive_aliases = true

This worked for me on Linux.

[bartfeenstra]

It appears that v0.990/v0.991 may not have addressed the entirety of the challenge that is recursive/cyclic types: https://github.com/python/mypy/issues/14219

[jamesbraza]

For those reading in posterity:

1. mypy==0.981 added --enable-recursive-alias: https://github.com/python/mypy/pull/13297
2. mypy==0.990 started a deprecation cycle for --enable-recursive-alias: https://github.com/python/mypy/pull/13852
3. mypy==1.7.0 completed the deprecation cycle, removing --enable-recursive-alias: https://github.com/python/mypy/pull/16346

So with mypy>=1.7, recursive type support is built into mypy ✅