search

hauntsaninja / mypy_primer

Run mypy and pyright over millions of lines of code
MIT License
55 stars 29 forks source link

Add support for type checking with mypy plugins #121

Closed flaeppe closed 3 days ago

flaeppe commented 2 months ago

~I'm not totally sure if this approach always works or if there's stuff that I'm missing, but it seems to work. I'll try to explain myself here.~

~The change utilises pip's --target(docs) argument to point out where to install a project's additional dependencies, passed in via pip_cmd. This replaces the virtual env creation for each project, since that shouldn't be necessary (I think?).~

~In addition to what's mentioned above~ we start setting the PYTHONPATH env variable(docs), to point out an additional search path for module files, namely the site-packages directory ~with the additional dependencies~ inside the project's virtualenv.

That should allow discovery of project dependencies while keeping the installed mypy versions independent and reusable.

Outdated --- All this _isn't_ included, but I was thinking that it _should_ be possible to use `pip install --target` for both mypy installations as well, instead of a virtual env. That leads to having 1 python/pip executable, 1 directory for `new_mypy`, 1 directory for `old_mypy` and then all project dependency installations. Although I'm unsure what `--python-executable=` does and if it could be affected. I'm also not sure how to handle compiled mypy. Essentially what'll happen with the above changes is that we'll have a directory structure like e.g. ```bash repo-dir/ ├── new_mypy │   └── bin │   └── mypy ├── old_mypy │   └── bin │   └── mypy ├── projects │   ├── A │   └── B └── python └── bin └── python ``` We run mypy with something corresponding to ``` PYTHONPATH="repo-dir/new_mypy:repo-dir/projects/A" repo-dir/new_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ... PYTHONPATH="repo-dir/old_mypy:repo-dir/projects/A" repo-dir/old_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ... ```

Here's some example debug output:

...
/tmp/mypy_primer/projects/_django-choicefield_venv/bin/pip install . django-stubs pytest         in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/new_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/new_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/old_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/old_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
/tmp/mypy_primer/new_mypy/venv/bin/mypy on django-choicefield took 4.96s
/tmp/mypy_primer/old_mypy/venv/bin/mypy on django-choicefield took 4.98s
...
flaeppe commented 2 months ago

Thanks for the review, I'm gonna try to move back to using project specific venvs, see if I can get it to work.

The global env was added to have a python/pip binary to reach to when I removed the project specific venvs.

I'll see if I can get a reasonable assert in there too, perhaps just something naive like looking for bin/mypy in a project specific venv is enough.

flaeppe commented 2 months ago

I've updated to align with your suggestions. Let me know if there's anything else I've missed or should update.

Though I can't say I follow why each project needs its own environment. I was thinking it would be preferable to have a virtualenv for each mypy installation and then do a "distributed" install for each project (pip install --target) and via PYTHONPATH= "mount" in the relevant dependencies for each project run. But I might definitely miss something.

hauntsaninja commented 3 days ago

Thanks again for the PR, sorry that I lost track of it! This developed a few merge conflicts, I fixed in 83721d5. I also altered the approach slightly, I use a .pth file to append to sys.path, so that project venv will never cause conflicts with mypy and its dependencies.