search

microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.34k stars 1.19k forks source link

pythonTerminalEnvVarActivation shadows pyenv PATH #23227

Open kbehlers opened 7 months ago

kbehlers commented 7 months ago

Behaviour

When the pythonTerminalEnvVarActivation experiment is active, I can no longer use pyenv-virtualenv commands for activating and deactivating venvs in the VSCode terminal. Instead the interpreter selected in the Python extension always stays activated in the terminal.

The pyenv-virtualenv commands give the appearance of working (they change my zsh command prompt showing the active venv) but running python commands like python --version or pip freeze show that the python interpreter being used isn't the one shown in my command prompt but instead the one selected using the Python extension in VSCode.

It seems possibly related to how pythonTerminalEnvVarActivation prepends to PATH.

Steps to reproduce:

  1. Make sure settings.json has 
    "python.experiments.optInto": ["pythonTerminalEnvVarActivation"]
  2. Install pyenv and pyenv-virtualenv
  3. Install two versions of python with pyenv (e.g. pyenv install 3.10 pyenv install 3.9) to make it easier to see when the wrong venv is active
  4. Create a virtualenv for each version of python 
    pyenv virtualenv 3.9 first_example_env
pyenv virtualenv 3.10 second_example_env
  5. Choose first_example_env as the python interpreter in VSCode using Python: Select Interpreter
  6. Open a VSCode terminal
  7. echo $PATH (this shows vscode specific paths when the experiment is enabled)
  8. python --version (should say Python 3.9.*)
  9. pyenv deactivate (doesn't end up deactivating as it should but gives the appearance of doing so by changing the command prompt)
  10. pyenv activate second_example_env (changes the command prompt, doesn't actually change the venv though)
  11. python --version (should say Python 3.10. but still says Python 3.9. because the environment didn't really switch)
  12. Use Python: Select Interpreter and choose second_example_env
  13. Reload the VSCode terminal
  14. Repeat above steps, except this time second_example_env will be active no matter what

If I instead add:

"python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"]

to settings.json, I can activate and deactivate venvs using pyenv-virtualenv just as I would expect, where the venv actually switches in the terminal.

Diagnostic data

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

``` 2024-04-11 11:24:39.228 [info] Prepending environment variable PATH in collection with /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin: {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-04-11 11:24:39.228 [info] Setting environment variable VIRTUAL_ENV in collection to /Users/username/.pyenv/versions/3.9.17/envs/first_example_env {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-04-11 11:24:39.228 [info] Prepending environment variable PS1 in collection with (first_example_env) {"applyAtShellIntegration":true,"applyAtProcessCreation":false} 2024-04-11 11:24:39.228 [error] Failed to initialize deactivate script /bin/zsh [Error: "/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh/envVars.txt" file not created at Timeout. (/Users/username/.vscode/extensions/ms-python.python-2024.4.1/out/client/extension.js:2:271175) at listOnTimeout (node:internal/timers:569:17) at process.processTimers (node:internal/timers:512:7)] ```

anthonykim1 commented 7 months ago

hello @kbehlers Thanks for filing the issue. Is this only happening with pyenv?? Is it fine with .venv

kbehlers commented 7 months ago

@anthonykim1 Hi! This is only with pyenv, specifically pyenv-virtualenv.

Standard .venv doesn't have the issue, and looking into it that is probably because the source .venv/bin/activate command prepends the venv to PATH itself..

pyenv-virtualenv seems to be relying on the existing shim already in the PATH, rather than prepending the venv each time, so the shadowing occurs because the VS Code experiment seems to prepend the venv from the interpreter selected in the Python extension.

Exploring the behavior a bit more, if I first source deactivate instead of pyenv deactivate then that clears the venv from the interpreter selected in the Python extension from PATH and I can use pyenv-virtualenv normally again.

anthonykim1 commented 7 months ago

Hello @kbehlers Thanks much for further looking into this.

I think it can be one or either of two problems:

  1. I'm also thinking about the shadowing scenarios of prepending possbily causing shadowing of pyenv adding to existing shim already in PATH. I believe we prepend to PATH all the time. When you take a look at your $PATH after you try to activate pyenv and see pyenv venv prompt, does your PATH contain pyenv related code?

  2. My second guess is the way we "deactivate" for pyenv. I will need to further look into deactivate scenario for pyenv. (Im going to try to figure out if python extension does sort of different deactivation for pyenv compared to traditional .venv and if that is correctly working.

Good to hear that there is at least some workaround if you source deactivate first.

I'll let you know if I get update on this.

anthonykim1 commented 7 months ago

Related: https://github.com/microsoft/vscode-python/pull/21906 Seems to be the PR where we started to use prepend to PATH. With its purpose of attempting to avoid replacing. since pyenv asks their user to manipulate PATH in init script.

anthonykim1 commented 6 months ago

@kbehlers can you share what your PATH looks like? We are trying to figure out which deactivation script is running when yousource deactivate vs. pyenv deactivate

kbehlers commented 6 months ago

@anthonykim1 Sure!

Opted out from the experiment

/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin

Opted into the experiment

/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/Cellar/pyenv-virtualenv/1.2.1/shims:/Users/username/.pyenv/shims:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh:/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin

You can see when opted-in it has some interesting prepend and append behavior

anthonykim1 commented 6 months ago

Thanks much for providing PATH. Greatly appreciated! The prepending is expected as: https://github.com/microsoft/vscode-python/pull/21906 and https://github.com/microsoft/vscode-python/pull/22905

Im not able to repro the behavior on my machine exactly but I do get "pyenv-virtualenv: no virtualenv has been activated." when I try pyenv deactivate when opting into the experiment. But then that still maintains Python from Pyenv virtualenv. Only when I source deactivate, it seems to bring back to "correct" Python.

Still lost on why prepending would "not allow" pyenv deactivate to work because your pyenv shim is still technically in your PATH even after Python extension prepend our deactivate script in the PATH.

kbehlers commented 6 months ago

@anthonykim1 Ahh, I see so if I am following https://github.com/microsoft/vscode-python/pull/21906 and https://github.com/microsoft/vscode-python/pull/22905 to it sounds like

PATH=<activated_full_path><activated_full_path><original_path>

might be expected then but I do want to highlight my path is actually ending up as 

PATH=<activated_full_path><activated_full_path><original_path><activated_full_path>

with the two prepended activated paths and a third appended activated_path. Probably not relevant to the problem, still a little odd to me though.

Going back to why prepending would "not allow" pyenv deactivate to work, I've been digging into this a bit more.

Built-in venv deactivate manipulates the PATH.

pyenv-virtualenv deactivate manipulates some env vars like VIRTUAL_ENV and PYENV_VIRTUAL_ENV but importantly it does not manipulate path. You can verify this by echo $PATH before and after a pyenv activate first_example_env and a pyenv deactivate.

The way pyenv uses its shims, it expects to do all python routing internally. As with pyenv-virtualenv, pyenv also does not manipulate PATH, instead as part of the installation instructions you modify .zshrc to add the shim to PATH upfront.

It was really helpful for me to understand the behavior by repeating my Steps to Reproduce, but checking PATH, PYENV_VERSION, VIRTUAL_ENV, and PYENV_VIRTUAL_ENV after each step.

As I understand it this means by having the venv force prepended to the PATH, pyenv deactivate will never be able to remove that, only a source deactivate will. Every time a python command comes in it finds the matching executable in the venv that was force prepended and never gets to the pyenv-vritualenv or pyenv shim so they can do an internal search against their activated and/or known python versions.

Out of the two paths appended

/Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh
:
/Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin

it isn't the /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh that is the problem, it is the /Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin, and that can be proven by removing only /Users/username/.pyenv/versions/3.9.17/envs/first_example_env/bin from the path while keeping /Users/username/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/zsh in place.

Hopefully that is helpful in trying to decide how to adapt the experiment for compatibility with pyenv.

dfuchs-dev commented 6 months ago

I'm experiencing a related issue. The appended paths that @kbehlers pointed out are causing problems with conda environments too. Here's what I observed after selecting my_conda_env via Python: Select Interpreter:

  1. While opted out of pythonTerminalEnvVarActivation, open a new terminal session, and echo the PATH:

    $ echo $PATH
/opt/anaconda3/envs/my_conda_env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin

  2. Now conda deactivate the session. The updated PATH does not contain any references to my_conda_env, and python is inaccessible, which is expected:

    
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin

$ which python python not found


3. Opt back into pythonTerminalEnvVarActivation, open a new terminal session, and echo the PATH. Note the extra `my_conda_env` second from last:

$ echo $PATH /opt/anaconda3/envs/my_conda_env/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/envs/my_conda_env/bin:/opt/anaconda3/condabin


4. Now `conda deactivate` the session. The updated PATH still contains `my_conda_env` and python is accessible when it should not be:

$ echo $PATH /usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/envs/my_conda_env/bin:/opt/anaconda3/condabin

$ which python /opt/anaconda3/envs/my_conda_env/bin/python



---

As a slight aside, even when opted out, the PATH in VS Code is still a bit different from what I get in a native terminal on my Mac.
 - VS Code, conda deactivated: `/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/anaconda3/condabin`
 - Native, conda deactivated: `/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin`

Since it's just condabin being reordered, I think it's fine, though a bit odd.