search

typeddjango / pytest-mypy-plugins

pytest plugin for testing mypy types, stubs, and plugins
https://pypi.org/project/pytest-mypy-plugins/
MIT License
105 stars 24 forks source link

Testing changes and caching behaviour #144

Closed delfick closed 2 months ago

delfick commented 8 months ago

Hello,

I'm not sure if I'm missing something obvious, but is there patterns for testing how my plugin interacts with mypy caching for both mypy and dmypy when code is changed?

So say I make a test case with code in a particular way and it passes mypy. I want a test that show it still passes mypy if I change that code in a particular way. Same with dmypy.

sobolevn commented 8 months ago

Right now this is not supported, but PR for this would be really welcomed! Mypy test cases have a similar design, here: https://github.com/python/mypy/blob/master/test-data/unit/check-incremental.test

delfick commented 8 months ago

mmkay, I thought so. Thanks. I will add this to the list of things I want to look at (no guarantees on a timeline :p)

delfick commented 6 months ago

So on this, once you're happy with https://github.com/typeddjango/pytest-mypy-plugins/pull/152 and that's merged, I have one last prefactor lined up (https://github.com/delfick/pytest-mypy-plugins/compare/schema/split-from-execution...delfick:pytest-mypy-plugins:separation/use-fixtures, will have an explanation of what it's doing when I make it a PR) and then finally the changes to make it possible to specify followups - https://github.com/delfick/pytest-mypy-plugins/compare/separation/use-fixtures...delfick:pytest-mypy-plugins:scenarios/followups

Though I have a problem I haven't quite being able to figure out where I change a file and then mypy doesn't see the change to that file. But if I do the same thing outside of pytest it does see the change. It's very weird.

edit: using the power of sleeping on it, I fixed the problem https://github.com/delfick/pytest-mypy-plugins/commit/89f2d5277c22872205616469065037e79b1a04e9

delfick commented 6 months ago

yay, after that current PR and the next two, I think I have all the changes in a state that is ready to use, was able to make this test on my extension on django-stubs https://github.com/delfick/extended-mypy-django-plugin/actions/runs/8981236513/job/24666367526 to show one of the things I need to fix that was slowing me down so much (an edge case where an app is removed from INSTALLED_APPS but is still present on the file system)

delfick commented 6 months ago

@sobolevn hello, I was wondering, I do want to eventually upstream these changes to this project, but my changes are quite extensive and your time for reviewing appears to be limited. Do you mind if I make my fork into it's own package in the meantime?

delfick commented 2 months ago

@sobolevn Hello, just thought I'd let you know that I ended up creating a fork of this plugin with all of my changes

https://pytest-typing-runner.readthedocs.io/en/latest/

The documentation still needs quite a bit of improvement tbh, but I've ran out of time to devote to that for now.

It currently only supports mypy but it shouldn't be difficult to make it possible to make it support running pyright or any other type checker. And makes it possible and easy to create scenarios where we test what happens when the code changes against a warm cache. It also is built around making it possible to write these tests like any other pytest test rather than constraining to a yaml definition (ultimately it provides a typing_scenario_runner fixture, which can be configured by a number of pytest fixtures so it can be customized within any normal pytest scope).

I've also created an example of how it could be used in django-stubs without major surgery to those tests if you're interested in that - https://github.com/typeddjango/django-stubs/compare/master...delfick:django-stubs:using-pytest-typing-runner?expand=1

An example of using it to write tests with followup scenarios can be found in https://github.com/delfick/extended-mypy-django-plugin/tree/main/tests

Either way, I want to say thanks for pytest-mypy-plugins! It gave me a massive head start with that work, I really appreciate everything you do around mypy and django stubs!