pylsp-mypy v0.5.4 crashes the entire pylsp diagnostics

[yaegassy]

Description

I am getting an error in v0.5.4 of "pylsp-mypy". Due to this error, it seems that no diagnostic messages are sent for pylsp as a whole.

This does not seem to be a problem with v0.5.2.

Repro

Create a venv dedicated to pylsp and install pylsp-mypy in it.

mkdir -p /tmp/pylsp
cd /tmp/pylsp
python3 -m venv venv
source venv/bin/activate
pip install python-lsp-server[all] pylsp-mypy

pip list:

(venv) pylsp$ pip list
Package            Version
------------------ -------
astroid            2.9.0
autopep8           1.6.0
flake8             4.0.1
isort              5.10.1
jedi               0.18.1
lazy-object-proxy  1.6.0
mccabe             0.6.1
mypy               0.910
mypy-extensions    0.4.3
parso              0.8.3
pip                21.2.4
platformdirs       2.4.0
pluggy             1.0.0
pycodestyle        2.8.0
pydocstyle         6.1.1
pyflakes           2.4.0
pylint             2.12.2
pylsp-mypy         0.5.4
python-lsp-jsonrpc 1.0.0
python-lsp-server  1.3.3
rope               0.22.0
setuptools         58.1.0
snowballstemmer    2.2.0
toml               0.10.2
typing_extensions  4.0.1
ujson              4.3.0
wrapt              1.13.3
yapf               0.31.0

Open the python file in an appropriate directory. I use vim.

mkdir -p /tmp/pylsp-mypy-check
cd pylsp-mypy-check
vim check.py

Error

The pylsp log shows that pylsp-mypy is causing the error. This error causes pylsp itself to not return any diagnostics messages such as flake8 to the LSP client.

It looks like it's trying to detect mypy, but it can't and is giving an error.

2021-12-14 14:42:26,565 JST - INFO - pylsp_mypy.plugin - lint settings = {} document.path = /private/tmp/pylsp-mypy-check/check.py is_saved = True
2021-12-14 14:42:26,565 JST - INFO - pylsp_mypy.plugin - executing mypy args = ['--show-column-numbers', '/private/tmp/pylsp-mypy-check/check.py', '--incremental', '--follow-imports', 'silent']
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/threading.py", line 1371, in run
    self.function(*self.args, **self.kwargs)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pylsp/_utils.py", line 33, in run
    return func(*args, **kwargs)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pylsp/python_lsp.py", line 299, in lint
    flatten(self._hook('pylsp_lint', doc_uri, is_saved=is_saved))
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pylsp/python_lsp.py", line 156, in _hook
    return hook_handlers(config=self.config, workspace=workspace, document=doc, **kwargs)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_hooks.py", line 265, in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 80, in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 327, in traced_hookexec
    return outcome.get_result()
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_result.py", line 33, in from_call
    result = func()
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_manager.py", line 324, in <lambda>
    lambda: oldcall(hook_name, hook_impls, kwargs, firstresult)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 60, in _multicall
    return outcome.get_result()
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_result.py", line 60, in get_result
    raise ex[1].with_traceback(ex[2])
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pluggy/_callers.py", line 39, in _multicall
    res = hook_impl.function(*args)
  File "/private/tmp/pylsp/venv/lib/python3.10/site-packages/pylsp_mypy/plugin.py", line 193, in pylsp_lint
    completed_process = subprocess.run(
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 501, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 966, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1842, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'mypy'

Misc

As a side note, if you install pylsp or pylsp-mypy in the venv of the project you are trying to edit, and then run that pylsp, you will not have the problem.

I'm sure there are a lot of people who have installed pylsp outside of the project folder, and in that case, I'm sure they are experiencing the same problem.

[Richardk2n]

For now revert to stable version.I will investigate asap.I have a likely cause in mind. (We had a change to better support venvs, this might have broken something.) Will probably take two to three evenings to fix. If I cannot reproduce I'll contact you again, otherwise assume that I'm working on it.

[Richardk2n]

I am not able to reproduce this with my editor of choice (spyder with python 3.9.9 though as my OS has not yet updated) and don't really know how to work with vim.

Can you tell me if "mypy check.py" works after activating the venv? Also "which mypy" would be interesting.

What OS are you on?

The steps to reproduce you posted above are not entirely accurate, are you sure this is what you did?

Also, if you could point me at an easy way to configure vim to use pylsp output, I will test that as well.

[yaegassy]

I use a Vim plugin called "coc.nvim". I developed a plugin called coc-pylsp for coc.nvim by myself and use coc-pylsp.

This problem of the whole pylsp diagnostics crashing occurs when mypy is not installed in the project.

The point of this problem is that "pylsp + pylsp-mypy" is installed outside of the project's venv.

It is very good that mypy is detected in the project (including venv), but if the detection fails, it might be better to catch the exception and skip it without error, or fall back to mypy installed with pylsp-mypy

[yaegassy]

Created a PR. https://github.com/Richardk2n/pylsp-mypy/pull/28

[Richardk2n]

Ok, I am very sorry, I had misunderstood, which venv is active, because I have little experience using venvs.

If I understood you right, pylsp-mypy gets executed with a python executable that does not belong to the currently active venv? (Is this considered save given, that things could break, if this python code relies on the variables being set correctly or subprocesses being executed in the same environment?)

So from what I gather, as it currently stands using a project venv to allow mypy to follow imports correctly only works if mypy is installed in said venv and uses the mypy installed in the venv. If mypy is not installed in the project venv but it is active while pylsp-mypy gets called with another executable, the plugin will (understandably) crash.

The best fix would be to fall back to the api call that existed before.

And I should probably recommend to the pylsp devs, that one plugin crashing should not crash the others.

[yaegassy]

If I understood you right, pylsp-mypy gets executed with a python executable that does not belong to the currently active venv?

@Richardk2n Yes, you are correct in that understanding.

Prepare a venv environment dedicated to pylsp, and use the pylsp installed there for all projects in common.

Since we are not installing in a global python environment, the mypy command that pylsp-mypy depends on does not exist on our PATH.

Some projects (different venvs for pylsp) may not use mypy. In that case, the mypy command does not exist, and you are getting a crash situation.

Probably "issue 26" is the same problem, since it is a pipx installation. https://github.com/Richardk2n/pylsp-mypy/issues/26

The best fix would be to fall back to the api call that existed before.

You can of course use whatever adjustment method you think is best for you. My PR is closed. 🙇

[Richardk2n]

Can you test if it is fixed in the newest release? It is likely fixed.

[yaegassy]

I have confirmed that the problem has been adjusted. This issue is closed.