microsoft / pyright

Static Type Checker for Python
Other
13.11k stars 1.4k forks source link

PEP 696: Incorrect argument inference when using `TypeVar` with `default` #8801

Closed Daraan closed 3 weeks ago

Daraan commented 3 weeks ago

Describe the bug

When using PEP 696 TypeVar(..., default=) with Unions that hide the args pyright reports some false positives as it does not track the __args__ correctly.

Code Repro

from typing_extensions import Concatenate, ParamSpec, TypeAlias, TypeVar, Callable

_D = TypeVar("_D", bound=bool, default=bool)
_PD = ParamSpec("_PD", default=...)
_ARG = TypeVar("_ARG", bound=float, default=float)

Alias1 : TypeAlias = Callable[Concatenate[_ARG, _PD], _D]
Alias2 : TypeAlias = Callable[Concatenate[str, _PD], _D]

Combo : TypeAlias = Alias1 | Alias2

# False Positive
# ERROR: Object of type Combo is not subscriptable Pylance: reportIndexIssue
Example: TypeAlias = Combo[int, ..., bool]

print(Combo.__parameters__)
# (~_ARG, ~_PD, ~_D)

Combo2: TypeAlias = Alias1 | Callable[Concatenate[float, _PD], _D]
print(Combo2.__parameters__)
# (~_ARG, ~_PD, ~_D)

# False Positive
# ERRORS:
# Arg1: Expected ParamSpec or ellipsis, think this is _PD
# Arg2: Thinks this is _D
# Arg3: Too many arguments for Combo2, expected 2 but received 3 Pylance: reportInvalidTypeForm
Example2: TypeAlias = Combo2[bool, [int, int], bool]

# False Negative or typing_extensions bug?
try:
    Alias1B = Alias1[int]  # pyright says this is correct
except TypeError as e:
    print(f"ERROR: {e}")
    # prints: ERROR: Too few arguments for typing.Callable[typing.Concatenate[~_ARG, ~_PD], ~_D]

I am not sure if the false negative at the end is pyright or typing_extensions limitation. I would need to read the PEP again, but thought this is valid.

VS Code extension or command-line Pyright command line: 1.1.377; as well as Pylance for VSCode

erictraut commented 3 weeks ago

Pyright's behavior is correct here, so this isn't a bug.

Generic classes and type aliases referenced within a type alias definition are either explicitly or implicitly specialized. In your code sample, you are defining a type alias Combo and referencing generic type aliases Alias1 and Alias2, but you haven't provided explicit type arguments for either of these type aliases. That means they receive their implicit (default) type arguments. The type alias Combo is not a generic type alias and cannot be further specialized.

If your intent is for Combo to be a generic type alias, you need to include type variables scoped to that type alias:

Combo: TypeAlias = Alias1[_ARG, _PD, _D] | Alias2[_PD, _D]

Similarly for Combo2. This is a generic type alias, but it has only two type parameters: _PD and _D.

Daraan commented 3 weeks ago

Thank you very much for the clarification. I now try to figure out what is valid and what is a typing_extensions bug. Could you answer some follow up questions for me?

1) Do you consider TypeAliasType as a valid alternative/workaround here or also as invalid?

Combo = TypeAliasType("Combo", Alias1 | Alias2, type_params=(_ARG, _PD, _D))

Pyright does not report, that I get a runtime error is probably a typing_extensions with <Python 3.11 limitation.


  1. I do not have a python 3.13 installation; do I understand it correctly that its then a typing_extensions limitation/bug that the parameters are (~_ARG, ~_PD, ~_D) and not (~_PD, ~_D) and I therefore get runtime errors?
erictraut commented 3 weeks ago

Do you consider TypeAliasType as a valid alternative/workaround here or also as invalid?

The statement Combo = TypeAliasType("Combo", Alias1 | Alias2, type_params=(_ARG, _PD, _D)) defines a generic type alias with three type parameters, but none of those type parameters are used anywhere in the type definition, so it's not a very useful type alias definition. It would be similar to defining a type alias like this:

type Foo[T] = int

This type alias is generic and accepts a single type argument, but T isn't used anywhere in the type definition.

Pyright does not report, that I get a runtime error is probably a typing_extensions with <Python 3.11 limitation.

I'm able to run the following code without a runtime exception on Python 3.10 and the latest published version of typing_extensions.

from typing import Callable, Concatenate, TypeAlias
from typing_extensions import TypeAliasType, TypeVar, ParamSpec

_D = TypeVar("_D", bound=bool, default=bool)
_PD = ParamSpec("_PD", default=...)
_ARG = TypeVar("_ARG", bound=float, default=float)

Alias1 : TypeAlias = Callable[Concatenate[_ARG, _PD], _D]
Alias2 : TypeAlias = Callable[Concatenate[str, _PD], _D]

Combo = TypeAliasType("Combo", Alias1 | Alias2, type_params=(_ARG, _PD, _D))

print(Combo.__parameters__)

If I append the following line, I see an exception when running on Python 3.10 but not on 3.12. I presume this is what you're referring to.

Alias1B = Alias1[int]

I'm guessing this is a limitation in older runtime implementations of Callable. You could file a bug against the typing_extensions library and see if the maintainers are willing to patch this behavior to accommodate this usage in older versions of Python.

If your real use case involves using these type aliases only within type expressions, you can enclose them in quotes to avoid the problem.

Alias1B: TypeAlias = "Alias1[int]"