Can't manually set path to pylsp per-project

[daisylb]

Because it's common to use multiple installations of Python for different projects (either to use different Python versions, or different virtualenvs), and the recommended way to handle this with python-lsp-server is to install the server into the virtualenv itself: https://github.com/emacs-lsp/lsp-mode/issues/393#issuecomment-404949030

There doesn't seem to be a way for me to tell this plugin where to look for the pylsp binary, so the only way I can use this plugin is by making sure I have the right install for the project I want to work on in my PATH, then launching Chime by running /Applications/Chime.app/Contents/MacOS/Chime, which is… not ideal (and will probably prevent me from working on multiple Python projects at once, too).

[mattmassicotte]

First, thank you so much for taking the time to open this issue. This kind of feedback is sooo helpful.

This problem sounds fairly similar to what we had to do for Ruby. That extension jumps though a bunch of hoops to find the language server and also use the desired configuration. I think we can do a similar thing here. But, to get it right, I need to better understand what a good approach would be. This kind of logic is exactly what the extensions are for - the goal is to make things as smooth as possible for users so they do not need to think about it.

I saw in the linked issue someone suggest starting with something like this:

#!/bin/bash
PYLS=`pwd`/venv/bin/pyls
$PYLS $@

I'm not familiar with pyls, or python in general. Do you have any suggestions on what should be done to best locate the executable?

[daisylb]

There's a bunch of different approaches to Python environment isolation. The most basic are "virtual environments", which are an isolated copy of Python in a user-specified directory with their own site-packages directories (which is where stuff goes when you pip install something). Then there's other tools (virtualenvwrapper, pyenv-virtualenv, pipenv, poetry, flit, ...) that typically will create a virtualenv in a predictable (but tool-specific) place, but there are plenty of people still just using ordinary virtualenvs in whatever location they please.

All that to say, I would not solely rely on autodetecting the right Python installation to use. You could write some auto detection logic to set a sensible default, but some way to manually specify a path to an executable is going to be a necessity in some cases.

Setting auto detection aside, there are two ways to have users set things up:

1. Tell users that they need to install the language server themselves into their virtualenv, and then let them configure a path to a pyls binary.
2. Have your own bundled copy of the language server, let your users specify the path to their python binary, and then prepend it to the Python module search path (so, doing the equivalent of PYTHONPATH=path/to/extension/bundled-packages path/to/user/configured/python -m pyls, which will run the language server in an environment that contains whatever packages the user has installed and the packages you bundle, with the latter taking precedence).

Most editor integrations I've seen seem to prefer the former approach, but I think there's merit to the latter: there's no extra step to getting it working on a project, the user doesn't have to add extra packages to their virtualenv that might cause dependency conflicts or other issues in other cases, and you don't have to worry as much about compatibility between the language server and the plugin because you know what version you're bundling.

[mattmassicotte]

Again, just fantastic information. I still have quite a bit of learning to do to fully understand what to do here. But, your recommendations are great.

Option 2 is interesting. That's what is currently done for Go. It turned out to be completely infeasible for Ruby, because mis-matches between Ruby versions used by the language server and source made for terrible incompatibilities. I'm unsure if that would be an issue or not here. But, I think I'm leaning towards some simple auto-detection to start.

There's currently no mechanism for an extension to communicate that something needs to be installed/changed, and that's sorely missed in a number of situations. Definitely something that needs to be improved.