MyPy thinks that tests decorated with `@given()` have `Any` in their definitions

[webknjaz]

So I'm trying to make MyPy stricter and hit this case that $sbj trips one of the Any checks with disallow_any_decorated = true set in the config.

I've bisected what happened to this small repro:

# repro_test.py 
from hypothesis import example, given, strategies as st

@example(text_input='криївка')
@given(text_input=st.text())
def test_repro(text_input: str) -> None:
    assert text_input

pip install 'mypy[reports]' hypothesis pytest
[...]
Successfully installed attrs-24.2.0 hypothesis-6.112.2 iniconfig-2.0.0 lxml-5.3.0 mypy-1.11.2 mypy-extensions-1.0.0 packaging-24.1 pluggy-1.5.0 pytest-8.3.3 sortedcontainers-2.4.0 typing-extensions-4.12.2
[...]

$ mypy --config=/dev/null --disallow-any-decorated --html-report=mypy-cov-report/ repro_test.py
/dev/null: No [mypy] section in config file
repro_test.py:6: error: Type of decorated function contains type "Any" ("Callable[..., None]")  [misc]
Generated HTML report (via XSLT): mypy-cov-report/index.html
Found 1 error in 1 file (checked 1 source file)

$ echo $?
1

$ python3 -Im webbrowser mypy-cov-report/index.html

I tried removing @example() and @given(). And deleting just @given() makes the error go away. Deleting just @example() makes it pass.

It seems like @given needs its typing improved, but I don't see where. I was staring at https://github.com/HypothesisWorks/hypothesis/blob/83c22d9/hypothesis-python/src/hypothesis/core.py#L1523-L1525 for a bit and couldn't see it. And I tried extending https://github.com/HypothesisWorks/hypothesis/blob/83c22d9/hypothesis-python/src/hypothesis/internal/reflection.py#L654-L661 with f.__annotations__ = target.__annotations__ but that's a runtime thing, and it didn't help (although, maybe it's still useful to add it there).

Additionally, looking into the HTML coverage report, the line with @example is highlighted as red, which is the original thing I was trying to hit. The report has that this demo module has 14.29% imprecise typing, and I'm targeting 100% type coverage (uploading to Codecov helps surface this information, if somebody's curious). I could understand why since it wasn't giving an error and MyPy's coverage reporting is sometimes unobvious, but some googling revealed that apparently things that have Any in them contribute to reduced type coverage. When I realized this, I started toggling MyPy strictness settings until I found one that exposes this error in the console output, since the coverage report does not display this context currently.

I'm pretty sure some typing needs fixing in Hypothesis itself, around the places I mentioned. Anything I'm missing?

[webknjaz]

Now I think that the problem starts @ https://github.com/HypothesisWorks/hypothesis/blob/83c22d9/hypothesis-python/src/hypothesis/core.py#L1427-L1459 and continues @ https://github.com/HypothesisWorks/hypothesis/blob/83c22d9/hypothesis-python/src/hypothesis/core.py#L1466. But beyond that, I'm stuck so far.

[Zac-HD]

I think you're correct to focus on the overloads here; but the runtime stuff - only matters for Pytest, not MyPy.

@overload
def given(*_given_arguments: SearchStrategy[Any]) -> Callable[
    [Callable[..., Optional[Coroutine[Any, Any, None]]]],
    Callable[..., None],  # <-- the `...` here is your Any
]: ...

@overload
def given(**_given_kwargs: Union[SearchStrategy[Any], EllipsisType]) -> Callable[
    [Callable[..., Optional[Coroutine[Any, Any, None]]]],
    Callable[..., None],  # <-- the `...` here is your Any
]: ...

Looking at this code with fresh eyes, the basic challenge to type-annotated it correctly is that we accept some arguments (positional xor keyword) and a function, and then return a function which accepts exactly the arguments which were not yet provided.

For the positional-args case, I think using ParamSpec, Concatenate, and either the new TypeVarTuple or some overloads for various numbers of arguments can make this fully typed. Which is great, and I'd love to accept a PR doing that!

Unfortunately I think the corresponding changes for keyword arguments just aren't supported in Python's static type system yet - see e.g. https://github.com/python/typing/issues/1009 with some good PEP links (they want to add a keyword argument while we want to remove dynamically-specific kwargs, but the logic and initial blocker is similar enough).

---

Once we get @given(whatever) working we can look at the other decorators; I think it'll be a pretty similar process. The main wrinkle there is that @example(whatever) can be applied either before or after @given(whatever) and must be compatible, but the static type system definitely doesn't let you express two decorators colluding to apply a particular transformation on the paramspec exactly once between them...

[webknjaz]

I was playing with that initially, and it seemed to require touching nested functions too. I didn't safe my experiments, but I remember it got to a point where MyPy was claiming that the decorator returned an untyped function or something that I couldn't figure out. Maybe, I'll try again later.