Python: Automatically set virtualenv paths

[exhuma]

When using mypy as linter, it needs to know where to look for packages. This breaks when using virtual environments.

Manually starting vim with:

MYPYPATH=/path/to/venv/lib/pythonX.Y/site-packages vim

will make it work. But this is tedious and I often forget it.

It would be nice if ALE was able to detect the path and automatically set it before running mmypy.

[w0rp]

https://mypy.readthedocs.io/en/latest/config_file.html Try using mypy_path in the configuration file.

[exhuma]

This works, but is a bit error-prone. The path from the virtual-environment contains the python version. So if the system Python package is updated, and the venv is re-created the path will no longer work and the config file must be modified.

From what I gather, ALE auto-detects the current Python executable which makes it always works. Even if the env is re-created with a new Python version.

I have a task-runner which creates a development environment and added the following into that task now:

env_path = fab.local('./env/bin/python -c "import sys; '
                     'print(\':\'.join(sys.path[1:]))"', capture=True)
if write_mypy:
    with open('mypy.ini', 'w') as fptr:
        fptr.writelines(['[mypy]\n', 'mypy_path=%s\n' % env_path])
print('%s%s' % (clr.yellow('Possible value for MYPYPATH:'), env_path))

This is not perfect but is an acceptable workaround.

It would be nice though if ALE had a config option so allow setting MYPYPATH to sys.path in case it's not set.

[exhuma]

I just noticed something quite bad with my solution: The paths created by sys.path obviously contain paths which don't exist on the machines of collaborators. So the created mypy.ini cannot be added to revision control.

This file already exists in my project and contains project-level settings which I want to share with collaborators. So setting this via mypy.ini does not seem to be the best idea.

[w0rp]

You should be able to use a configuration file like this.

[mypy]

mypy_path = stubs

You can run mypy via a virtualenv executable so PYTHONPATH is pretty good, or run it via pipenv.

I don't use mypy myself in my Python project at the moment. I have found that it executes far too slowly, and there aren't nearly enough type definitions to make it useful. This project from Facebook may be more useful in the future: https://github.com/facebook/pyre-check

[exhuma]

I've actually started using mypy once I started using ALE. Even if mypy is slow, aler will make it will eventually show up. Which is great. I really love ALE for that.

In our team we have quite a large amount of Python libs which are written in-house and are type-checked. It was really worth the effort doing it (even though some things are not yet fully checked).

But I currently have difficulties with the virtualenv in vim (the reason for creating this ticket). pylint (which is also run via ale) seems to find the venv without issue. mypy doesn't.

And putting stubs into a separate folder in each project is far from practical. All our packages are PEP-561 compliant so writing stubs would be really nonsense. Currently I don't always activate the environment because I switch between different projects fairly frequently. I usually just do ./env/bin/python ..., ./env/bin/pylint ..., ./env/bin/mypy ... which works fine.

Except when running via ale.

[w0rp]

You may have more luck with pipenv or a Vim plugin that activates virtualenv. I probably can't explain how to configure it exactly right, but there's probably a way to do it.

[w0rp]

You may also have some luck with b:ale_command_wrapper if you need something really project specific.

[w0rp]

I think I know how to handle this. MYPYPATH will maybe need some special logic, but we should add a general g:ale_python_auto_virtualenv setting, or whatever name makes sense. It should probably be off by default as to not break current configurations people are comfortably using. One thing it can do is update PATH like so:

" This probably won't work on Windows, so don't do literally this.
let l:env_vars_to_pass_to_job = {'PATH': l:venv_dir . '/bin:' . $PATH}

This alone is enough to get pylsp to play nicely with its mypy plugin. I've tested it with my current Python codebase.

[w0rp]

@exhuma I have now set up ALE to automatically set PATH based on virtualenv detection so long as you set b:ale_python_auto_virtualenv or b:ale_python_auto_virtualenv to 1. Let me know if that fixes your issue.

[jocado]

Hi

I just found this issue, as I was investigating the lack of venv support the mypy seemed to have from Ale. But, I realised that the behaviour is the same with the CLI utility mypy too. Specifically, it doesn't seem to respect the PATH variable somehow in any standard way.

I can't completely explain the behaviour, but the module searching seems to be quite custom, and what I found was, to make the CLI util work in the venv, you had to point it at the python executable:

mypy --python-executable "${VIRTUAL_ENV}/bin/python3" {some-location}

The it seems to work with the modules installed in the venv.

I then tried various things to add that to the vimrc config via:

let g:ale_python_mypy_options = '--python-executable {blah}'

But even hard coding the full path there didn't seem to fix it when called from Ale.

For now, I'm resorting to this to stop it complaining, but still seems to leave type hint checking working. Possible, it won't work if the types are imported form another module, not sure.

let g:ale_python_mypy_options = '--ignore-missing-imports'

There's lots of info here about how module searching works. https://mypy.readthedocs.io/en/stable/running_mypy.html

Not sure if with the above information Ale could somehow support venv by setting a shell expanded version of --python-executable "${VIRTUAL_ENV}/bin/python3" automatically ?

[sebastianw]

I think I have the same issue. In my case mypy is installed globally via pipx in it's own virtual environment and not into every project/venv. To use it in a virtualenv I need to do mypy --python-executable "${VIRTUAL_ENV}/bin/python3" something.py.