annotations on `self` are not understood with `Protocol`s

[finite-state-machine]

Bug Report

When interpreting a given type as a given Protocol, annotations on self in the Protocol are apparently ignored.

To Reproduce

mypy-play.net

from __future__ import annotations

from typing import *

                          #━━━━━━━━━━━━━━━━━#
                          #  EXISTING CODE  #
                          #━━━━━━━━━━━━━━━━━#

class ConcreteClass:

    # here 'transform()' is used, but we might use '__neg__()' or
    # '__invert__()' in a similar way (so many built-in/third-party
    # classes would qualify as 'TransformAware')

    def transform(self) -> int:
        return id(self)

                           #━━━━━━━━━━━━━━━━#
                           #  LIBRARY CODE  #
                           #━━━━━━━━━━━━━━━━#

# this can be changed

Ico = TypeVar('Ico', covariant=True)        # input, covariant
Oco = TypeVar('Oco', covariant=True)        # output, covariant

class TransformAware(Protocol[Ico, Oco]):

    def transform(self: Ico) -> Oco: ...

                      #━━━━━━━━━━━━━━━━━━━━━━━━━#
                      #  DEMONSTRATE PROBLEMS:  #
                      #━━━━━━━━━━━━━━━━━━━━━━━━━#

I = TypeVar('I')                            # input
O = TypeVar('O')                            # output

def interpret_transform_aware(_: Type[TransformAware[I, O]]) -> Tuple[I, O]:
    i: I
    o: O
    return i, o

reveal_type(interpret_transform_aware(ConcreteClass))
        # actual output:
        #   note: Revealed type is "Tuple[<nothing>, builtins.int]"
        # expected output:
        #   note: Revealed type is "Tuple[ConcreteClass, int]"

                           #───────────────#
                           #  MOTIVATION:  #
                           #───────────────#

class TransformDriver(Generic[I, O]):

    def __init__(self, klass: Type[TransformAware[I, O]]): ...

    def transform_value(self, value: I) -> O:
        use_value = cast(TransformAware[I, O], value)
        return use_value.transform()

Expected Behavior

(See: inline comments.)

Interpreting ConcreteClass as TransformAware should correctly infer both type variables.

Actual Behavior

(See: inline comments.)

Mypy gives the type of Ico as <nothing>.

Your Environment

• Mypy version used: 1.6.1
• Mypy command-line flags: (none necessary)
• Mypy configuration options from mypy.ini (and other config files): (none necessary)
• Python version used: 3.8, 3.11

[finite-state-machine]

I did also try to work around this problem by omitting the Ico variable from TransformAware, but I couldn't get that working either.

https://mypy-play.net/?mypy=latest&python=3.11&gist=fc4531ce46437863136431ea1f0f3d91

from __future__ import annotations

from typing import *

class ConcreteClass:

    def transform(self) -> int:
        return id(self)

Ico = TypeVar('Ico', covariant=True)        # input, covariant
Oco = TypeVar('Oco', covariant=True)        # output, covariant

class TransformAware(Protocol[Oco]):

    def transform(self) -> Oco: ...

class TransformDriver(Generic[I, O]):

    I2 = TypeVar('I2', bound='TransformAware[O]')

    def __init__(self: TransformDriver[I2, O], klass: I2): ...

    def transform_value(self, value: I, /) -> O:
        use_value = cast(TransformAware[O], value)
        return use_value.transform()

I = TypeVar('I')                            # input
O = TypeVar('O')                            # output

def interpret_transform_aware(_: Type[TransformAware[O]]) -> O:
    o: O
    return o

reveal_type(interpret_transform_aware(ConcreteClass))
        # as expected:
        #   Revealed type is "builtins.int"

def interpret_transform_driver(_: TransformDriver[I, O]) -> Tuple[I, O]:
    i: I
    o: O
    return i, o

reveal_type(interpret_transform_driver(TransformDriver(ConcreteClass)))
        # actual output:
        #   Revealed type is "Tuple[I2`-1, <nothing>]"
        # expected output:
        #   Revealed type is "Tuple[ConcreteClass, int]"

The ideal solution would be to have TransformDrivers I parameter bound to TransformAware[O], but I can't see any way of doing that.

[erictraut]

The type of self must be type consistent with the class that defines the method. In your code, the type variable Ico is not consistent with TransformAware. If you add an upper bound on the type variable Ico (and similarly for I), this type checks fine.

Ico = TypeVar("Ico", covariant=True, bound="TransformAware")

[finite-state-machine]

Thanks, @erictraut!

I've modified the file as you suggest (IIUC), and while there are no errors, the type of ConcreteType when interpreted as a TransformAware remains TransformAware[<nothing>, int] rather than TransformAware[ConcreteType, int] as expected (line 59).

Instantiating TransformDriver does cause problems (see: inline, below).

(Any suggested workarounds would be most appreciated!)

mypy-play.net

from __future__ import annotations

from typing import *

                          #━━━━━━━━━━━━━━━━━#
                          #  EXISTING CODE  #
                          #━━━━━━━━━━━━━━━━━#

class ConcreteClass:

    # here 'transform()' is used, but we might use '__neg__()' or
    # '__invert__()' in a similar way (so many built-in/third-party
    # classes would qualify as 'TransformAware')

    def transform(self) -> int:
        return id(self)

                           #━━━━━━━━━━━━━━━━#
                           #  LIBRARY CODE  #
                           #━━━━━━━━━━━━━━━━#

# this can be changed

Ico = TypeVar('Ico', bound='TransformAware[Any, Any]', covariant=True)          # ← CHANGED
        # input, covariant
Oco = TypeVar('Oco', covariant=True)
        # output, covariant

class TransformAware(Protocol[Ico, Oco]):

    def transform(self: Ico) -> Oco: ...

                      #━━━━━━━━━━━━━━━━━━━━━━━━━#
                      #  DEMONSTRATE PROBLEMS:  #
                      #━━━━━━━━━━━━━━━━━━━━━━━━━#

I = TypeVar('I', bound='TransformAware[Any, Any]')          # input             # ← CHANGED
O = TypeVar('O')                                            # output

def interpret_transform_aware(_: Type[TransformAware[I, O]]) -> Tuple[I, O]:
    i: I
    o: O
    return i, o

reveal_type(interpret_transform_aware(ConcreteClass))
        # actual output:
        #   note: Revealed type is "Tuple[<nothing>, builtins.int]"
        # expected output:
        #   note: Revealed type is "Tuple[ConcreteClass, int]"

                           #───────────────#
                           #  MOTIVATION:  #
                           #───────────────#

class TransformDriver(Generic[I, O]):

    def __init__(self, klass: Type[TransformAware[I, O]]): ...

    def transform_value(self, value: I) -> O:
        use_value = cast(TransformAware[I, O], value)
        return value.transform()

cc_driver = TransformDriver(ConcreteClass)                                      # ← NEW ↓
        # error: Need type annotation for "cc_driver"  [var-annotated]
reveal_type(cc_driver)
        # actual output:
        #   Revealed type is "TransformDriver[Any, builtins.int]"
        # expected output:
        #   Revealed type is "TransformDriver[ConcreteClass, builtins.int]"

[erictraut]

What you're doing here is very unusual. I don't think it will work because of the way protocol matching is implemented in type checkers.

It's not clear to me why you're using a type annotation for the self parameter in your protocol class. The type of self should always be Self in a protocol. After all, a protocol is a structural type definition, so self takes on whatever type the protocol matches against.

Here's a simplified version of your code that avoids the use of a self annotation in the protocol.

from typing import Generic, Protocol, TypeVar

class ConcreteClass:
    def transform(self) -> int:
        return id(self)

Oco = TypeVar("Oco", covariant=True)

class TransformAware(Protocol[Oco]):
    def transform(self) -> Oco:
        ...

class TransformDriver(Generic[Oco]):
    def __init__(self, klass: type[TransformAware[Oco]]):
        ...

    def transform_value(self, value: TransformAware[Oco]) -> Oco:
        return value.transform()

cc_driver = TransformDriver(ConcreteClass)
reveal_type(cc_driver)

[finite-state-machine]

Thanks, @erictraut. I hadn't considered this strategy because transform_value() would accept any type of TransformAware[Oco] (rather than just instances of the klass passed to __init__()), but I think that will have to do (unless you have any other helpful ideas – not that you haven't already been very helpful!)