mypy plugin to ignore type-checking errors with pytest.raises

[Dreamsorcerer]

When writing tests for error cases using with pytest.raises(...), the code within the body will always generate a mypy error which must be ignored (if a project has sufficient static typing).

It would be great if pytest included a mypy plugin which can suppress the errors in these cases.

Example: https://github.com/aio-libs/aiohttp-session/blob/5f0ffbd7932e2ee256e0fb0ae4cdcad50d73cddc/tests/test_redis_storage.py#L346

[nicoddemus]

Hi @Dreamsorcerer,

the code within the body will always generate a mypy error which must be ignored

Not in general (in fact in almost all cases in my experience), example:

def foo(x: int) -> None:
    if x > 10: 
        raise ValueError

def test_foo_raises() -> None:
    with pytest.raises(ValueError):
        foo(20)

Seems to me here you are testing what the type checker is already testing for you, which I don't think it is necessary in a project which is sufficiently typed.

[Dreamsorcerer]

Yep, not in all cases then, but I see it frequently. Just because our project is typed, doesn't mean that users are doing type checking when using our library, so there are still runtime checks etc. which are covered by tests.

[nicoddemus]

Fair enough, that is indeed interesting for testing APIs.

I've changed the title to better reflect the request, thanks for writing!

[bluetech]

I don't think pytest is going to ship a mypy plugin. It's quite finicky and besides, there are other type checkers nowadays.

The problem you are describing is a more general one than pytest, although pytest.raises(TypeError) is a common manifestation. I once suggested adding a type: expect[arg-type] type comment for these cases but haven't had time to pursue it further. With this approach, you can annotate the line as expected to issue some type error, which then becomes a type-checking time test in itself (it will cause an error if the type checking error doesn't happen).

[Dreamsorcerer]

I think it would be better where we don't have to add comments everytime. A generic solution might be to have something in the return type of pytest.raises() that declares this automatically (in a similar vein to TypeGuard), and says that an error is expected in the with block.

[Dreamsorcerer]

But, it does feel like something that's quite specific to testing, so I'm not sure how useful this would be elsewhere, which is why I suggested a plugin for this use case. I guess unittest should also have this ideally, so maybe a plugin that is separate from pytest and works with both libraries would make more sense.

[Andrej730]

Met the similar issue - was wondering how I can create a test file that will ensure some cases will have typing issues. The solution for me was --warn-unused-ignores + # type: ignore [xxxx]

# run with --warn-unused-ignores

def test(a: int) -> None: ...
test("5") # type: ignore [arg-type]

def test1(a: str) -> None: ...
# Unnecessary "# type: ignore" comment
test1("5") # type: ignore