Wrong Python Binary Used for RPC Virtual Environment

[drgillis]

Summary

Elpy sometimes locates the wrong Python executable when creating the virtual environment. In my case, it finds the python3 binary in /usr/bin/ rather than /usr/local/bin, even though the latter is higher in my path (and in my exec-path variable).

The problem seems to be in the function elpy-rpc-get-virtualenv-path in elpy-rpc.el. The let binds exec-path to its reverse, which causes it to search lower priority directories before higher priority ones when (executable-find elpy-rpc-python-command) is run. I looked at the pull-request from when this code was added to elpy (#1682), and I believe that exec-path is being reversed here to locate a system executable in the event that a virtual environment is active.

One possible solution is to create a custom variable where the user can define what directory (or directories) they want to be searched for their python binary. This gives the user a fall-back solution if the wrong one is found after exec-path is reversed. I now see that the variable elpy-rpc-python-command can be used in this manner. However, this fact seems to be non-obvious.

As a short-term workaround, I ran (setq exec-path (append exec-path '("/usr/local/bin"))) before running (elpy-config).

Steps to reproduce

1. Set elpy-rpc-python-command to "python3".
2. Delete (or move) the ~/.emacs.d/elpy/ directory.
3. Create two dummy directories and put a different version of the python3 binary in each of them.
4. Add both of these dummy directories to your exec-path, keeping track of the order they are inserted.
5. Run (elpy-config).
6. Notice that the RPC Python is the version of Python in the latter of the two dummy directories in exec-path.

My configuration

OS

macOS 10.15

Result of (elpy-config)

Emacs.............: 28.0.50
Elpy..............: 1.35.0
Virtualenv........: None
Interactive Python: python3 3.9.4 (/usr/local/bin/python3)
RPC virtualenv....: rpc-venv (/Users/drg/.emacs.d/elpy/rpc-venv)
 Python...........: python3 3.7.3 (/Users/drg/.emacs.d/elpy/rpc-venv/bin/python3)
 Jedi.............: Not found
 Rope.............: Not found
 Autopep8.........: Not found
 Yapf.............: Not found
 Black............: Not found
Syntax checker....: flake8 (/usr/local/bin/flake8)

Elpy configuration in my init.el

None

[j3pic]

I've resorted to using the following shell script to fix Elpy on machines where I've just installed it:

#!/bin/sh

set -e

cd ~/.emacs.d/elpy
cat << EOT > /tmp/elpy-requirements.txt
annotated-types==0.5.0
autopep8==2.0.0
black==23.3.0
click==8.1.5
flake8==5.0.4
importlib-metadata==4.2.0
jedi==0.18.2
mccabe==0.7.0
mypy-extensions==1.0.0
packaging==23.1
parso==0.8.3
pathspec==0.11.1
platformdirs==3.8.1
pycodestyle==2.9.1
pydantic==2.0.3
pydantic_core==2.3.0
pyflakes==2.5.0
pytoolconfig==1.2.5
rope==1.9.0
tomli==2.0.1
typed-ast==1.5.5
typing_extensions==4.7.1
yapf==0.33.0
zipp==3.15.0
EOT

rm -rf rpc-venv
python -m venv rpc-venv

I extracted the dependencies from the output of pip freeze in a working Elpy virtualenv.