`Self` and `self` are not bound until after class definition, becoming `Any`/`Never` when passed through a decorator

[jace]

Bug Report

There is occasional use for a method that exhibits different behaviours when accessed via an instance vs the class. SQLAlchemy has one such example named hybrid_method (code). I was experimenting with a generic decorator for this pattern and found what appear to be two bugs in Mypy 1.7.0. The code works at runtime and passes type validation in Pyright 1.1.337 (barring a no-redef error).

First problem, given the following code pattern:

class Host:
    # T = TypeVar, P = ParamSpec, R = TypeVar
    @Decorator  # (Callable[Concatenate[T, P], R]) -> Decorator[Generic[T, P, R]]
    def method(self) -> Self:
        return self

    reveal_type(method)  #  T=Never, R=Any

Type T is self and R is Self, and both should be the same. The decorator has to be Generic[P, R] to return Callable[P, R] in __get__, but R then turns into Any. If the decorator is Generic[T, P, R], then R becomes Never and Host().method() errors on the self parameter: expected Never, got Host. reveal_type shows that self's type is lost within the class definition itself. Pyright processes it correctly.

Second problem, when the decorator implements the @prop.setter pattern to replace itself with a new object with different generic binds. Mypy raises a no-redef error, which can be type-ignored, but the revealed type continues to show the old generic binds. This problem goes away when the second method is given a different name.

To Reproduce

Here's sample code with a functioning decorator and three ways to use it. This code works at runtime and passes Pyright (barring one redef), but each of these versions has different errors or loss of type information in Mypy.

https://mypy-play.net/?mypy=latest&python=3.11&gist=03abdb6bef5ae78eea51d6ae07a0e778

Tested in Mypy 1.7, Pyright 1.1, Python 3.9 and 3.11

[jace]

I think Mypy's handling of self isn't completely wrong, even though it breaks typing via the decorator but works for built-ins like @property and @classmethod. The type of the self/cls parameter is only known at call time. The type assigned at definition time (host class's type or a subtype) is only the usual type for the usual descriptor-based access (cls.method or obj.method). Which means the problem is in the decorator's typing for its descriptor:

def __get__(self, obj: T | None, cls: type[T]) -> Callable[P, R]: ...

There is no indication that the Callable is typed T, so the Self type in R can't be linked to T. Maybe it can be done using a Protocol?

class WrappedMethod(Protocol[T, P, R]):
    def __call__(self: T, *args: P.args, **kwargs: P.kwargs) -> R: ...

class Decorator:
    ...
    def __get__(self, obj: T | None, cls: type[T]) -> WrappedMethod[T, P, R]: ...

[jace]

Okay, checked this up. Specifying a type on self in Protocol.__call__ makes it not callable, so dead-end there.

Pyright also correctly processes Self when the decorated method is accessed via a subclass. I tried the pre-Self method of a bound TypeVar but that doesn't work in both Pyright and Mypy:

class Decorator(Generic[P, R]):
    def __init__(self, fget: Callable[Concatenate[Any, P], R]): ...
    @overload
    def __get__(self, obj: None, cls: Type[T]) -> Callable[Concatenate[T, P], R]: ...
    @overload
    def __get__(self, obj: T, cls: Type[T]) -> Callable[P, R]: ...
    def __get__(self, obj: Optional[T], cls: Type[T]) -> Union[Callable[P, R], Callable[Concatenate[T, P], R]]: ...

ParentType = TypeVar('ParentType', bound='Parent')

class Parent:
    @Decorator
    def method_self(self, arg: str) -> Self:
        return self

    @Decorator
    def method_typevar(self: ParentType, arg: str) -> ParentType:
        return self

class Child(Parent):
    pass

reveal_type(Parent.method_self)
# Pyright: "(arg: str) -> Parent"  (correct)
# Mypy: "(arg: str) -> Any"

reveal_type(Parent.method_typevar)
# Pyright: "(arg: str) -> Any"
# Mypy:  "(arg: str) -> Any"

reveal_type(Child.method_self)
# Pyright: "(arg: str) -> Child" (correct)
# Mypy: "(arg: str) -> Any"

reveal_type(Child.method_typevar)
# Pyright: "(arg: str) -> Any"
# Mypy:  "(arg: str) -> Any"

PEP 673 – Self Type makes no mention of how Self should be processed in a generic decorator except for @classmethod (which is anyway special-cased in type checkers), so this looks like a gap in the spec.

The difference in how Pyright and Mypy bind Self comes out when calling an unbound method:

class Parent:
    def method(self) -> Self: return self

class Child(Parent): pass

s1 = Child()
s2 = Parent.method(s1)
reveal_type(s2)
# Runtime: Child
# Pyright: Parent (wrong)
# Mypy: Parent (correct)

So: Pyright is binding Self from the class or instance, which is usually right, but wrong when calling an unbound method, whereas Mypy is more accurately binding it from the self parameter, but that is lost in the decorator's __get__ signature. If only there was a way to add it back with something like MethodType[T, Callable[P, R]] (or with Protocol).

Related: #16558, cc: @erictraut and @AlexWaygood

[jace]

Test code: https://mypy-play.net/?mypy=latest&python=3.11&gist=e8de541c111bccf640907937f8794a1b

[erictraut]

Pyright is binding Self from the class or instance ... [which is] wrong when calling an unbound method

I think this result is still correct from a type perspective. In your example above, the runtime produces a value which is a Child instance, and this is a valid subtype of Parent, which is the type evaluated by pyright. In other words, this isn't a "wrong" type evaluation. It's perhaps not as narrow as you'd like, but that can't be helped if binding is done when it should be.

I think binding and partial specialization must be done when evaluating the attribute access expression (Parent.method), not as part of the call expression (Parent.method(s1)). This becomes clear when you consider generic classes. Consider if Parent were generic and accepted a single type parameter, as in the modified example below:

from typing import Generic, Self, TypeVar

T = TypeVar("T")

class Parent(Generic[T]):
    def method(self) -> Self:
        return self

class Child(Parent[int]):
    pass

s1 = Child()
s2 = Parent[str].method(s1) # Pyright: Type violation (correct), mypy: no error
reveal_type(s2)

Here, mypy's approach results in a clear false negative.

Binding rules are admittedly underspecified in the Python typing spec. This would be a good topic for the newly-formed typing council to clarify so we can get consistent behavior between type checkers.

[jace]

I think binding and partial specialization must be done when evaluating the attribute access expression

I agree. This has the nice side effect that the generic decorator is simpler because it doesn't have to bind on type.

In my first code sample above, I made the mistake of defining self type as invariant because that's the default for TypeVar, but self has to be covariant. I had no idea what these terms meant last month, and I imagine it's going to cause grief to many others writing type annotations for the first time.