search

typeddjango / pytest-mypy-plugins

pytest plugin for testing mypy types, stubs, and plugins
https://pypi.org/project/pytest-mypy-plugins/
MIT License
100 stars 26 forks source link

test failures are really really difficult to read #153

Open asottile opened 1 month ago

asottile commented 1 month ago

here's an example failure I had to debug recently:

$ pip install . -qq --no-deps && python -m pytest tests -k test_emits_error_for_unresolved_managers
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-8.3.1, pluggy-1.5.0 -- /tmp/django-stubs/venv/bin/python
cachedir: .pytest_cache
rootdir: /tmp/django-stubs
configfile: pytest.ini
plugins: mypy-plugins-3.1.2, shard-0.1.2
collected 415 items / 414 deselected / 1 selected                              
Running 1 items in this shard: tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers

tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers FAILED

=================================== FAILURES ===================================
___________________ test_emits_error_for_unresolved_managers ___________________
/tmp/django-stubs/tests/typecheck/managers/test_managers.yml:495: 
E   pytest_mypy_plugins.utils.TypecheckAssertionError: Invalid output: 
E   Actual:
E     ...
E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E     myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:54: note: Revealed type is "Any" (diff)
E     myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:56: note: Revealed type is "Any" (diff)
E     myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E     myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E     myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:65: note: Revealed type is "Any" (diff)
E     myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:67: note: Revealed type is "Any" (diff)
E     myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E   Expected:
E     ...
E     myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects"  [django-manager-missing] (diff)
E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
E     myapp/models:40: note: Revealed type is "django.db.models.manager.Manager[myapp.models.Booking]" (diff)
E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
E     myapp/models:53: note: Revealed type is "def () -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:54: note: Revealed type is "Any" (diff)
E     myapp/models:55: note: Revealed type is "def (*args: Any, **kwargs: Any) -> myapp.models.UnknownQuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:56: note: Revealed type is "Any" (diff)
E     myapp/models:57: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E     myapp/models:58: note: Revealed type is "myapp.models.Booking" (diff)
E     myapp/models:59: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:60: note: Revealed type is "builtins.list[myapp.models.Booking]" (diff)
E     myapp/models:64: note: Revealed type is "def () -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:65: error: "RelatedManager[Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:65: note: Revealed type is "Any" (diff)
E     myapp/models:66: note: Revealed type is "def (*args: Any, **kwargs: Any) -> django.db.models.query.QuerySet[myapp.models.Booking, myapp.models.Booking]" (diff)
E     myapp/models:67: error: "QuerySet[Booking, Booking]" has no attribute "custom"  [attr-defined] (diff)
E     myapp/models:67: note: Revealed type is "Any" (diff)
E     myapp/models:68: note: Revealed type is "Union[myapp.models.Booking, None]" (diff)
E   Alignment of first line difference:
E     E: myapp/models:32: error: Could not resolve manager type for "myapp.models...
E     A: myapp/models:36: note: Revealed type is "django.db.models.manager.Manage...
E                      ^
=========================== short test summary info ============================
FAILED tests/typecheck/managers/test_managers.yml::test_emits_error_for_unresolved_managers - 
====================== 1 failed, 414 deselected in 3.89s =======================

the "Alignment of first line difference" part is "useful" at least, but every line is marked as (diff) -- I ended up copying Actual and Expected to a file and using diff -u:

$ diff -u expected actual
--- expected    2024-07-25 15:31:06.184912437 -0400
+++ actual  2024-07-25 15:31:08.888821824 -0400
@@ -1,4 +1,3 @@
-E     myapp/models:32: error: Could not resolve manager type for "myapp.models.InvisibleUnresolvable.objects"  [django-manager-missing] (diff)
 E     myapp/models:36: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
 E     myapp/models:37: note: Revealed type is "django.db.models.manager.Manager[myapp.models.User]" (diff)
 E     myapp/models:39: note: Revealed type is "myapp.models.UnknownManager[myapp.models.Booking]" (diff)
@@ -6,7 +5,7 @@
 E     myapp/models:42: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
 E     myapp/models:43: note: Revealed type is "myapp.models.UnknownManager[myapp.models.TwoUnresolvable]" (diff)
 E     myapp/models:44: note: Revealed type is "django.db.models.manager.Manager[myapp.models.TwoUnresolvable]" (diff)
-E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.InvisibleUnresolvable]" (diff)
+E     myapp/models:46: note: Revealed type is "myapp.models.UnknownManager[myapp.models.AbstractUnresolvable]" (diff)
 E     myapp/models:47: note: Revealed type is "django.db.models.manager.Manager[myapp.models.InvisibleUnresolvable]" (diff)
 E     myapp/models:49: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)
 E     myapp/models:50: note: Revealed type is "django.db.models.fields.related_descriptors.RelatedManager[myapp.models.Booking]" (diff)

much much more readable! and at least as a programmer I'm used to looking at unified diffs and would have been able to spot the difference much quicker!

sobolevn commented 1 month ago

Yes, this is really hard. It took me several minutes just to review one line change (I had to count lines separately, because lines in IDE do not match lines in main.py example).

I am open to suggestions.