typeddjango / pytest-mypy-plugins

pytest plugin for testing mypy types, stubs, and plugins
https://pypi.org/project/pytest-mypy-plugins/
MIT License
99 stars 26 forks source link

Output broken for multiline messages #66

Closed zero323 closed 2 years ago

zero323 commented 2 years ago

Given following test case

- case: break_following
  main: |
    from typing import overload, TypeVar

    T = TypeVar("T", int, str)

    @overload
    def f(x: int) -> int: ...
    @overload
    def f(x: str) -> str: ...
    def f(x: T) -> T:
      return x

    reveal_type(f([1, 2, 3]))  # N: Revealed type is "Any"

will get message as follows:

E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Expected:
E     <45 (diff)
E   Actual:
E     main:12: error: No overload variant of "f" matches argument type "List[int]" (diff)
E     main:12: note:     def f(x: int) -> int       (diff)
E     main:12: note:     def f(x: str) -> str       (diff)
E     main:12: note: Possible overload variants:    (diff)
E     main:12: note: Revealed type is "Any"         (diff)
E   
E   Alignment of first line difference:
E     E: main:12: note: Revealed type is "Any"...
E     A: main:12: error: No overload variant of "f" matches argument type "List[i...
E    

while mypy will return

main.py:13: error: No overload variant of "f" matches argument type "List[int]"
main.py:13: note: Possible overload variants:
main.py:13: note:     def f(x: int) -> int
main.py:13: note:     def f(x: str) -> str
main.py:13: note: Revealed type is "Any"

Please note that

E     main:14: note: Possible overload variants:    (diff)

is misplaced compared to mypy output.

It happens, because sorting key includes actual error message:

https://github.com/typeddjango/pytest-mypy-plugins/blob/f7b249d0e40c25e2979adfbdb804a2e489072837/pytest_mypy_plugins/utils.py#L209

so sort is not stable within (fname, line_number).

Environment: