google / pytype

A static type analyzer for Python code
https://google.github.io/pytype
Other
4.77k stars 279 forks source link

Methods with unknown decorators have self typed Any #1381

Open eltoder opened 1 year ago

eltoder commented 1 year ago
from typing import Any

# This comes from a third-party library.
def decorator() -> Any: ...

class C:
    @decorator()
    def method(self) -> None:
        reveal_type(self)  # revealed as Any

In this example pytype does not know the precise type of @decorator which comes from a 3rd-party library. In my case this is @given from hypothesis. pytype assumes that self has type Any, which lead to both false negatives (misspelled attributes are not caught) and false positives (assertIsInstance doesn't narrow types).

Other than manually annotating type of self in all cases or improving the type of decorator in the 3rd-party library, is there another solution? For comparison, mypy reveals the type as C in this example.

rchen152 commented 1 year ago

Ugh, yeah, this has been a known issue for a while. I'm afraid there's no good workaround other than the ones you've mentioned (annotate self or make the type signature of decorator known). Annotating self is probably easier, although if you want to give pytype the type signature, you can at least do it in your own file rather than having to touch the 3rd-party library, with something like this:

if TYPE_CHECKING:
  def decorator(f: _T) -> _T: ...
else:
  from wherever import decorator
eltoder commented 1 year ago

The @given decorator is pretty hard to type precisely. It takes a bunch of strategies and passes a value from each to the decorated function. (It also does the same for keyword arguments, which I'll ignore.) I think this requires TypeVarTuple with a non-standard extension to apply a type constructor:

T = TypeVar("T")
Ts = TypeVarTuple("Ts")

def given(*args: *Map[SearchStrategy, Ts]) -> Callable[[Callable[[T, *Ts], None]], Callable[[T], None]]:
    ...

(Map here applies SearchStrategy to every type in the type tuple and returns the resulting type tuple. Also, here it is used in reverse: it requires that types of all varargs are SearchStrategy and extracts the type arguments into the type tuple Ts.)

I guess I can cheat and do

T = TypeVar("T")
P = ParamSpec("P")
R = TypeVar("R")

def given(*args: SearchStrategy[Any]) -> Callable[[Callable[Concatenate[T, P], R]], Callable[[T], R]]:

which is slightly better than the current signature.

If you have simpler ideas, please let me know :-)

eltoder commented 1 year ago

Actually, I tried this, and it did not work. If I do

T = TypeVar("T")

def given() -> Callable[[T], T]:
    ...  # pytype: disable=bad-return-type

class Test:
    @given()
    def test_foo(self) -> None:
        reveal_type(self)  # revealed as Test

reveal_type(Test.test_foo)  # revealed as Callable[[Any], None]

The type of self is preserved inside test_foo (but not outside). But anything even slightly more complicated breaks this:

T = TypeVar("T")

def given() -> Callable[[Callable[[T], None]], Callable[[T], None]]:
    ...  # pytype: disable=bad-return-type

class Test:
    @given()
    def test_foo(self) -> None:
        reveal_type(self)  # revealed as Any

reveal_type(Test.test_foo)  # revealed as Any