get_type_hints change in behavior for Annotated with from future import annotations 3.9+

[hmc-cs-mdrissi]

Bug report

from __future__ import annotations

from typing import Sequence, Union
from typing_extensions import Annotated, get_type_hints

Config = Union[str, Sequence["Config"]]
class Foo:
 x: Annotated[Config, "hi"]

print(Config)
print(get_type_hints(Foo)["x"])

The output prints are different in 3.8.13 vs 3.9.13/3.10.4/3.11.0b1. In 3.8.7 the output is,

typing.Union[str, typing.Sequence[ForwardRef('Config')]]
typing.Union[str, typing.Sequence[ForwardRef('Config')]]

In 3.9.13/3.10.4/3.11.0b1 the output is,

typing.Union[str, typing.Sequence[ForwardRef('Config')]]
typing.Union[str, typing.Sequence[typing.Union[str, typing.Sequence[ForwardRef('Config')]]]]

get_type_hints is doing 1 type expansion on forward reference. There's second inconsistency here which is that Annotated changes output in 3.9+. I'd expect get_type_hints(Foo, include_extras=False) to have same type with or without Annotated. If I drop Annotated and instead test

Config = Union[str, Sequence["Config"]]
class Foo:
 x: Config

print(Config)
print(get_type_hints(Foo)["x"])

the 1 type expansion goes away and output is same as 3.8.7 output.

The from __future__ import annotations is important or you can also manually quote and do x: "Config".

The exact behavior is niche and would be understandable if this was documented as undefined implementation detail similar to how cross module aliases are awkward to work with at runtime.

Expansion affected runtime internal serialization tool I was using and I ended up needing to do some workarounds to fix tests. I haven't tried testing if pydantic/similar libraries handle this case well or not. Expansion also affects documentation produced by a tool like sphinx-autodoc-typehints.

Your environment I'm on mac, although hope this isn't platform specific. I also tried testing from typing import Annotated instead of typing_extensions for 3.9+ and it didn't resolve inconsistency. My first guess is _eval_type is where this behavior change comes from. Newest version I tested up to was 3.11.0b1.

[picnixz]

In 3.12+, we would likely use the type keyword for Config, namely:

from __future__ import annotations

from collections.abc import Sequence
from typing import Annotated, get_type_hints

type Config = str | Sequence[Config]
class Foo:
    x: Annotated[Config, "hi"]
    y: Config

print(Config)
print(get_type_hints(Foo)['x'])
print(get_type_hints(Foo)['y'])

All of them print as Config, where there is from __future__ import annotations or not (at least on 3.12.2). With include_extras, the second print shows typing.Annotated[Config, 'hi']. Should this issue be considered as resolved for 3.12+? (unfortunately, <3.12 won't get any bugfix...).

[AlexWaygood]

In 3.12+, we would likely use the type keyword for Config, namely:

Should this issue be considered as resolved for 3.12+? (unfortunately, <3.12 won't get any bugfix...).

The old style of creating type aliases (without the type keyword) will likely continue to be widely used until Python 3.11 goes end of life. I'd still see this as worth fixing on 3.12+ until we get to that point.

[picnixz]

I'll see whether I can do something for this (I don't have much on my backlog now so I can try for a few hours).

[picnixz]

It seems that the Annotated has nothing to do with the double expansion:

import typing

Config = str | list["Config"]

class X:
    x: Config

print(typing.get_type_hints(X)['x'])
# str | list[str | list[ForwardRef("Config")]]

versus (with from __future__ import annotations)

from __future__ import annotations
import typing

Config = str | list['Config']

class X:
    x: Config

print(typing.get_type_hints(X)['x'])
# str | list[ForwardRef("Config")]

I think we can do something by detecting whether the type is a recursive type or not and whether a forward reference needs to be re-expanded, but it might be tricky and I don't have all corner cases in my head. I don't want to make something slower. I'll need to put down some tricks on paper.