microsoft / vscode-python

Python extension for Visual Studio Code
https://aka.ms/pvsc-marketplace
MIT License
4.31k stars 1.18k forks source link

Terminal environment variables loaded by extension are corrupted #23604

Open aatle opened 4 months ago

aatle commented 4 months ago

Type: Bug

Behaviour

In the terminal under specific circumstances, some environment variables are loaded with invalid values because of backslashes, causing certain commands to fail.

(project-py3.12)
abcde@MY-PC MINGW64 C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code
$ git --help
fatal: unable to access 'C:\Users?bcde/.config/git/config': Invalid argument
(project-py3.12) 
abcde@MY-PC MINGW64 C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code
$ echo $HOME
C:\Usersbcde

It is expected that the environment variable values would not have special escape sequences from backslashes in Windows paths.

Steps to reproduce:

VS Code Python extension loading poetry virtual environment on Git Bash on Windows

  1. Use Windows (11). The name of your User Account, the name seen in C:\Users, should start with lowercase a, e.g. "abcde". (Some other characters probably also work for this: b,e,f,n,r,t,v.)
  2. Have Git installed. Have python installed (I am using 3.12.4). Have poetry installed (1.8.3 used, but probably any version) using pip.
  3. Create a new folder. Open it with VS Code and create a pyproject.toml file with the following contents:
    
    [tool.poetry]
    name = "project"
    version = "0.1.0"
    description = ""
    authors = []

[tool.poetry.dependencies] python = "^3.10"

[build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"

4. Open the terminal, use Git Bash as the default (instead of Command Prompt, PowerShell).
5. Exit VS Code and reopen the project. Kill the terminal and reopen it. The PS1 prompt will now have something like `(project-py3.12)`, see above.
6. Git commands such as `git --help` fail. Possibly some other specific commands also fail. The `HOME`, `PATH`, and `EXEPATH` environment variables have invalid values.

<!--
**After** creating the issue on GitHub, you can add screenshots and GIFs of what is happening. Consider tools like https://www.cockos.com/licecap/, https://github.com/phw/peek or https://www.screentogif.com/ for GIF creation.
-->

<!-- **NOTE**: Please do provide logs from Python Output panel. -->
# Diagnostic data

<details>

<summary>Output for <code>Python</code> in the <code>Output</code> panel</summary>

(User account name replaced with `abcde`.)

<p>

2024-06-12 18:06:58.366 [info] Experiment 'pythonaacf' is active 2024-06-12 18:06:58.366 [info] Experiment 'pythonCreateEnvOnPipInstall' is active 2024-06-12 18:06:58.366 [info] Experiment 'pythonRecommendTensorboardExt' is active 2024-06-12 18:06:58.366 [info] Experiment 'pythonTerminalEnvVarActivation' is active 2024-06-12 18:06:58.366 [info] Experiment 'pythonTestAdapter' is active 2024-06-12 18:06:58.366 [info] Default formatter is set to charliermarsh.ruff for workspace c:\Users\abcde\development\reproduce 2024-06-12 18:06:58.366 [info] Starting Environment refresh 2024-06-12 18:06:58.366 [info] Starting Environment refresh 2024-06-12 18:06:58.366 [info] > hatch env show --json 2024-06-12 18:06:58.366 [info] cwd: . 2024-06-12 18:06:58.366 [info] > poetry env list --full-path 2024-06-12 18:06:58.366 [info] cwd: . 2024-06-12 18:06:59.770 [info] Environments refresh finished (event): 1467 milliseconds 2024-06-12 18:06:59.770 [info] Environments refresh paths discovered: 1467 milliseconds 2024-06-12 18:06:59.771 [info] Environments refresh finished (event): 1467 milliseconds 2024-06-12 18:06:59.771 [info] Environments refresh paths discovered: 1467 milliseconds 2024-06-12 18:06:59.772 [info] Environment refresh took 1468 milliseconds 2024-06-12 18:06:59.772 [info] Environment refresh took 1469 milliseconds 2024-06-12 18:06:59.780 [info] > pyenv which python 2024-06-12 18:06:59.780 [info] cwd: . 2024-06-12 18:06:59.826 [info] Running installed packages checker: { id: 'C:\USERS\ABCDE\APPDATA\LOCAL\PYPOETRY\CACHE\VIRTUALENVS\PROJECT-3JZSXIOL-PY3.12\SCRIPTS\PYTHON.EXE', sysPrefix: 'C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12', envType: 'Poetry', envName: 'project-3JzsXiol-py3.12', envPath: 'C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12', path: 'C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts\python.exe', architecture: 1, sysVersion: undefined, version: { raw: '3.12.4', major: 3, minor: 12, patch: 4, build: [], prerelease: [ 'final', '0' ] }, displayName: "Python 3.12.4 ('project-3JzsXiol-py3.12')", detailedDisplayName: "Python 3.12.4 ('project-3JzsXiol-py3.12': Poetry)", type: 'Virtual' } c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\python_files\installed_check.py c:\Users\abcde\development\reproduce\pyproject.toml 2024-06-12 18:06:59.826 [info] > ~\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts\python.exe ~.vscode\extensions\ms-python.python-2024.8.0\python_files\installed_check.py .\pyproject.toml 2024-06-12 18:06:59.828 [info] Python interpreter path: ~\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts\python.exe 2024-06-12 18:06:59.856 [info] > . ~/AppData/Local/pypoetry/Cache/virtualenvs/project-3JzsXiol-py3.12/Scripts/activate && echo 'e8b39361-0157-4923-80e1-22d70d46dee6' && python ~/.vscode/extensions/ms-python.python-2024.8.0/python_files/printEnvVariables.py 2024-06-12 18:06:59.856 [info] shell: bash 2024-06-12 18:07:00.080 [info] > ~\AppData\Local\Programs\Python\Python312\python.exe ~/.vscode/extensions/ms-python.python-2024.8.0/python_files/printEnvVariables.py 2024-06-12 18:07:00.080 [info] shell: bash 2024-06-12 18:07:00.137 [info] Setting environment variable EXEPATH in collection to C:\Program Files\Git\bin {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.137 [info] Setting environment variable HOME in collection to C:\Users\abcde {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.137 [info] Setting environment variable MSYS in collection to disable_pcon {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.137 [info] Setting environment variable MSYSTEM in collection to MINGW64 {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.137 [info] Prepending environment variable PATH in collection to c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\python_files\deactivate\bash;C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\abcde\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\Git\cmd;C:\Users\abcde\AppData\Local\Programs\Python\Python312\Scripts;C:\Users\abcde\AppData\Local\Programs\Python\Python312;C:\Users\abcde\AppData\Local\Microsoft\WindowsApps;C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\bin;C:\users\abcde.local\bin; {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable PLINK_PROTOCOL in collection to ssh {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Prepending environment variable PS1 in collection with (project-py3.12) {"applyAtShellIntegration":true,"applyAtProcessCreation":false} 2024-06-12 18:07:00.138 [info] Setting environment variable PWD in collection to C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable PYTHONIOENCODING in collection to utf-8 {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable PYTHONUNBUFFERED in collection to 1 {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable TERM in collection to xterm-256color {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable VIRTUAL_ENV in collection to C:/Users/abcde/AppData/Local/pypoetry/Cache/virtualenvs/project-3JzsXiol-py3.12 {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Setting environment variable VIRTUAL_ENV_PROMPT in collection to project-py3.12 {"applyAtShellIntegration":true,"applyAtProcessCreation":true} 2024-06-12 18:07:00.138 [info] Send text to terminal: C:/Users/abcde/AppData/Local/Programs/Python/Python312/python.exe c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\python_files\printEnvVariablesToFile.py c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\python_files\deactivate\bash\envVars.txt 2024-06-12 18:07:00.797 [info] Starting Jedi language server for reproduce. 2024-06-12 18:07:30.143 [error] Failed to initialize deactivate script C:\Program Files\Git\bin\bash.exe [Error: "c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\python_files\deactivate\bash\envVars.txt" file not created at Timeout. (c:\Users\abcde.vscode\extensions\ms-python.python-2024.8.0\out\client\extension.js:2:273846) at listOnTimeout (node:internal/timers:573:17) at process.processTimers (node:internal/timers:514:7)]


</p>
</details>

<details>

<summary>Terminal Environment Changes data (open terminal, hover over <code>bash - Microsoft VS Code</code>, click <code>Show environment contributions</code>)</summary>

Note that the git extension was able to load its environment variables correctly.

<p>

```md
# Terminal Environment Changes

## Extension: vscode.git

Enables the following features: git auth provider

- `GIT_ASKPASS=c:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass.sh`
- `VSCODE_GIT_ASKPASS_NODE=C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\Code.exe`
- `VSCODE_GIT_ASKPASS_EXTRA_ARGS=`
- `VSCODE_GIT_ASKPASS_MAIN=c:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass-main.js`
- `VSCODE_GIT_IPC_HANDLE=\\.\pipe\vscode-git-c39b414ee2-sock`

## Extension: ms-python.python

Activated environment for `~\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts\python.exe`

- `EXEPATH=C:\Program Files\Git\bin`
- `HOME=C:\Users\abcde`
- `MSYS=disable_pcon`
- `MSYSTEM=MINGW64`
- `PATH=c:\Users\abcde\.vscode\extensions\ms-python.python-2024.8.0\python_files\deactivate\bash;C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\abcde\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\Git\cmd;C:\Users\abcde\AppData\Local\Programs\Python\Python312\Scripts;C:\Users\abcde\AppData\Local\Programs\Python\Python312;C:\Users\abcde\AppData\Local\Microsoft\WindowsApps;C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\bin;C:\users\abcde\.local\bin;${env:PATH}`
- `PLINK_PROTOCOL=ssh`
- `PS1=(project-py3.12) ${env:PS1}`
- `PWD=C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code`
- `PYTHONIOENCODING=utf-8`
- `PYTHONUNBUFFERED=1`
- `TERM=xterm-256color`
- `VIRTUAL_ENV=C:/Users/abcde/AppData/Local/pypoetry/Cache/virtualenvs/project-3JzsXiol-py3.12`
- `VIRTUAL_ENV_PROMPT=project-py3.12`

Three corrupted environment variables

Bash ```bash $ echo $HOME C:\Usersbcde $ echo $PATH c:\Usersbcde\.vscodetensions\ms-python.python-2024.8.0\python_files\deactivatash;C:\Usersbcde\AppData\Local\pypoetry\Cache irtualenvs\project-3JzsXiol-py3.12\Scripts;C:\Program Files\Git\mingw6in;C:\Program Files\Git\usin;C:\Usersnthin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell 1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\Git/c/Users/abcde/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/abcde/bin:/c/Users/abcde/.vscode/extensions/ms-python.python-2024.8.0/python_files/deactivate/bash:/c/Users/abcde/AppData/Local/pypoetry/Cache/virtualenvs/project-3JzsXiol-py3.12/Scripts:/mingw64/bin:/usr/bin:/c/Users/abcde/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/cmd:/c/Users/abcde/AppData/Local/Programs/Python/Python312/Scripts:/c/Users/abcde/AppData/Local/Programs/Python/Python312:/c/Users/abcde/AppData/Local/Microsoft/WindowsApps:/c/Users/abcde/AppData/Local/Programs/Microsoft VS Code/bin:/c/users/abcde/.local/bin:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/WINDOWS/System32/OpenSSH:/c/Program Files (x86)/Windows Kits/10/Windows Performance Toolkit:/cmd:/c/Users/abcde/AppData/Local/Programs/Python/Python312/Scripts:/c/Users/abcde/AppData/Local/Programs/Python/Python312:/c/Users/abcde/AppData/Local/Microsoft/WindowsApps:/c/Users/abcde/AppData/Local/Programs/Microsoft VS Code/bin:/c/users/abcde/.local/bin:/usr/bin/vendor_perl:/usr/bin/core_perl $ echo $EXEPATH C:\Program Files\Giin ``` Python (shows escape sequences) ```bash $ python Python 3.12.4 (tags/v3.12.4:8e8a4ba, Jun 6 2024, 19:30:16) [MSC v.1940 64 bit (AMD64)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import os >>> os.environ["HOME"] 'C:\\Users\x07bcde' >>> os.environ["EXEPATH"] 'C:\\Program Files\\Git\x08in' >>> os.environ["PATH"] 'c;C:\\Users\x07bcde\\.vscode\x1bxtensions\\ms-python.python-2024.8.0\\python_files\\deactivate\x08ash;C;C:\\Users\x07bcde\\AppData\\Local\\pypoetry\\Cache\x0birtualenvs\\project-3JzsXiol-py3.12\\Scripts;C;C:\\Program Files\\Git\\mingw64\x08in;C;C:\\Program Files\\Git\\usr\x08in;C;C:\\Users\x07bcde\x08in;C;C:\\WINDOWS\\system32;C;C:\\WINDOWS;C;C:\\WINDOWS\\System32\\Wbem;C;C:\\WINDOWS\\System32\\WindowsPowerShell\x0b1.0;C;C:\\WINDOWS\\System32\\OpenSSH;C;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit;C;C:\\Program Files\\Git\\c\\Users\\abcde\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\local\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Users\\abcde\\bin;C:\\Users\\abcde\\.vscode\\extensions\\ms-python.python-2024.8.0\\python_files\\deactivate\\bash;C:\\Users\\abcde\\AppData\\Local\\pypoetry\\Cache\\virtualenvs\\project-3JzsXiol-py3.12\\Scripts;C:\\Program Files\\Git\\mingw64\\bin;C:\\Program Files\\Git\\usr\\bin;C:\\Users\\abcde\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\WINDOWS\\System32\\OpenSSH;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit;C:\\Program Files\\Git\\cmd;C:\\Users\\abcde\\AppData\\Local\\Programs\\Python\\Python312\\Scripts;C:\\Users\\abcde\\AppData\\Local\\Programs\\Python\\Python312;C:\\Users\\abcde\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\abcde\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\users\\abcde\\.local\\bin;C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0;C:\\WINDOWS\\System32\\OpenSSH;C:\\Program Files (x86)\\Windows Kits\\10\\Windows Performance Toolkit;C:\\Program Files\\Git\\cmd;C:\\Users\\abcde\\AppData\\Local\\Programs\\Python\\Python312\\Scripts;C:\\Users\\abcde\\AppData\\Local\\Programs\\Python\\Python312;C:\\Users\\abcde\\AppData\\Local\\Microsoft\\WindowsApps;C:\\Users\\abcde\\AppData\\Local\\Programs\\Microsoft VS Code\\bin;C:\\users\\abcde\\.local\\bin;C:\\Program Files\\Git\\usr\\bin\\vendor_perl;C:\\Program Files\\Git\\usr\\bin\\core_perl' ```

Notes

It would seem that the Python extension is attempting to load environment variables with values that include backslashes because of Windows paths. But somewhere in the process, a few of the backslashes are being interpreted as escape sequences. For example \v becomes a vertical tab and \b becomes a backspace. The interpretation aligns with how a .env file interprets backslashes. See C Escape Sequences.\ If the user does not have a user account name that starts with a valid escape sequence character, only PATH is corrupted, and not enough to be immediately noticeable.\

A workaround is adding "python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"] to settings.json.

Extension version: 2024.8.0 VS Code version: Code 1.90.0 (89de5a8d4d6205e5b11647eb6a74844ca23d2573, 2024-06-04T19:33:54.889Z) OS version: Windows_NT x64 10.0.22631 Modes:

User Settings

``` languageServer: "Jedi" experiments • optOutFrom: ["pythonTerminalEnvVarActivation"] ```

Installed Extensions |Extension Name|Extension Id|Version| |---|---|---| |Azure Repos|ms-vscode.azure-repos|0.38.0| |Even Better TOML|tamasfe.even-better-toml|0.19.2| |GitHub Pull Requests|GitHub.vscode-pull-request-github|0.88.1| |GitHub Repositories|GitHub.remotehub|0.62.0| |JavaScript Debugger|ms-vscode.js-debug|1.90.0| |JavaScript Debugger Companion Extension|ms-vscode.js-debug-companion|1.1.2| |Mypy|matangover.mypy|0.3.1| |Python|ms-python.python|2024.8.0| |Python Debugger|ms-python.debugpy|2024.6.0| |Remote Repositories|ms-vscode.remote-repositories|0.40.0| |Ruff|charliermarsh.ruff|2024.26.0| |Table Visualizer for JavaScript Profiles|ms-vscode.vscode-js-profile-table|1.0.9|
System Info |Item|Value| |---|---| |CPUs|11th Gen Intel(R) Core(TM) i7-11700F @ 2.50GHz (16 x 2496)| |GPU Status|2d_canvas: enabled
canvas_oop_rasterization: enabled_on
direct_rendering_display_compositor: disabled_off_ok
gpu_compositing: enabled
multiple_raster_threads: enabled_on
opengl: enabled_on
rasterization: enabled
raw_draw: disabled_off_ok
skia_graphite: disabled_off
video_decode: enabled
video_encode: enabled
vulkan: disabled_off
webgl: enabled
webgl2: enabled
webgpu: enabled| |Load (avg)|undefined| |Memory (System)|15.77GB (6.93GB free)| |Process Argv|--crash-reporter-id 937dcfed-06f3-46d6-bf2d-b1f57e7b00a9| |Screen Reader|no| |VM|0%|
A/B Experiments ``` vsliv368cf:30146710 vspor879:30202332 vspor708:30202333 vspor363:30204092 vscorecescf:30445987 vscod805cf:30301675 binariesv615:30325510 vsaa593:30376534 py29gd2263:31024239 c4g48928:30535728 azure-dev_surveyone:30548225 962ge761:30959799 pythongtdpath:30769146 welcomedialog:30910333 pythonidxpt:30866567 pythonnoceb:30805159 asynctok:30898717 pythontestfixt:30902429 pythonregdiag2:30936856 pythonmypyd1:30879173 h48ei257:31000450 pythontbext0:30879054 accentitlementsc:30995553 dsvsc016:30899300 dsvsc017:30899301 dsvsc018:30899302 cppperfnew:31000557 dsvsc020:30976470 pythonait:31006305 jchc7451:31067544 chatpanelc:31048052 dsvsc021:30996838 f3je6385:31013174 pythoncenvpt:31062603 a69g1124:31058053 dvdeprecation:31068756 pythonprt:31056678 dwnewjupytercf:31046870 newcmakeconfigv2:31071590 26j00206:31048877 ```
anthonykim1 commented 3 months ago

Hi @aatle Thanks for filing this issue.

Can you opt out from the new terminal environment variable experiment for now and see if the problem persist? You can do this by adding "python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"], into your settings.json

aatle commented 3 months ago

As I have said in the original post, that setting is indeed a workaround and fixes the problem. Unfortunately, it is actually not a perfect solution because it also introduces a different problem.

(Unrelated to the setting), the Python extension will sometimes still activate the poetry virtual environment but decide to only edit (and corrupt) the PATH variable, instead of editing a bunch of variables. Not sure why.

The Environment changes show the differences: Project 1: Only `PATH` corrupted, and the prompt (PS1) stays the same: ```markdown ## Extension: ms-python.python Activated environment for `~\AppData\Local\pypoetry\Cache\virtualenvs\pyriak-WneeXSsL-py3.12\Scripts\python.exe` - `PATH=c:\Users\abcde\.vscode\extensions\ms-python.python-2024.8.1\python_files\deactivate\bash;C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\pyriak-WneeXSsL-py3.12\Scripts;${env:PATH}` ``` Project 2: reproducible example, prompt changes to show virtual environment, more variables corrupted: ```markdown ## Extension: ms-python.python Activated environment for `~\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts\python.exe` - `EXEPATH=C:\Program Files\Git\bin` - `HOME=C:\Users\abcde` - `MSYS=disable_pcon` - `MSYSTEM=MINGW64` - `PATH=c:\Users\abcde\.vscode\extensions\ms-python.python-2024.8.1\python_files\deactivate\bash;C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\project-3JzsXiol-py3.12\Scripts;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\abcde\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit;C:\Program Files\Git\cmd;C:\Users\abcde\AppData\Local\Programs\Python\Python312\Scripts;C:\Users\abcde\AppData\Local\Programs\Python\Python312;C:\Users\abcde\AppData\Local\Microsoft\WindowsApps;C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\bin;C:\users\abcde\.local\bin;${env:PATH}` - `PLINK_PROTOCOL=ssh` - `PS1=(project-py3.12) ${env:PS1}` - `PWD=C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code` - `PYTHONIOENCODING=utf-8` - `PYTHONUNBUFFERED=1` - `TERM=xterm-256color` - `VIRTUAL_ENV=C:/Users/abcde/AppData/Local/pypoetry/Cache/virtualenvs/project-3JzsXiol-py3.12` - `VIRTUAL_ENV_PROMPT=project-py3.12` ```

For the latter Project 2, the setting indeed fixes things, no environment variables are corrupted, and the extension does not contribute to terminal environment.\ \ For the former, the setting seems to corrupt PATH even more, so sed and other commands are not found, rendering the terminal almost useless. The environment changes show that the Python extension is still contributing PATH, but has a different value.

Project 1 terminal environment changes, with setting enabled ```markdown ## Extension: ms-python.python - `PATH=C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\pyriak-WneeXSsL-py3.12\Scripts;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files\Git\cmd;C:\Users\abcde\AppData\Local\Programs\Python\Python312\Scripts\;C:\Users\abcde\AppData\Local\Programs\Python\Python312\;C:\Users\abcde\AppData\Local\Microsoft\WindowsApps;C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\bin;c:\users\abcde\.local\bin` ```
github-actions[bot] commented 2 months ago

Hey @anthonykim1, this issue might need further attention.

@aatle, you can help us out by closing this issue if the problem no longer exists, or adding more information.

aatle commented 2 months ago

This issue still exists, and makes all git commands fail on some of my poetry projects.

anthonykim1 commented 2 months ago

but decide to only edit (and corrupt) the PATH variable, instead of editing a bunch of variables. Not sure why.

I believe this is part of the Terminal Env Var experiment as we prepend PATH and calculate other environment variables from user. https://github.com/microsoft/vscode-python/pull/22905

When you say "corrupted" PATH, are you looking the prepending pattern?

aatle commented 2 months ago

Yes, in PATH, I am mainly looking at the prepended paths that have backslashes in them.

My original post isn't entirely clear, I'll clarify and summarize what I think is happening:

I am using Windows, poetry virtual environment, and Git Bash terminal. From my understanding, the opt-out experimental feature edits and calculates certain environment variables. The feature has one problem: in the environment variables edited, some backslash literals (often present in Windows paths) become special escape sequences e.g. \v, \n. Only certain characters such as a and n can form escape sequences with backslash, so not all backslashes are corrupted.

So, paths in the edited environment variables such as C:\Program Files\Git\bin become C:\Program Files\Giin where there is a \b backspace character in it. This obviously renders those paths useless/corrupted. This problem is worse if the user's $HOME variable is also affected. C:\Users\abcde (calculated) -> C:\Usersbcde with special \a character. (Without Python extension env var experiment, it would be /c/Users/abcde, using forward slashes.)

So, I think that there is a backslash issue somewhere in the process of parsing, editing, and exporting environment variable strings.

If I opt out of this feature using the setting, then some of my other environments, which look slightly different for some reason, get even more corrupted such that basic commands like sed are not found and PATH is corrupted.

anthonykim1 commented 1 month ago

@aatle Apologies for late response and thanks for further explanation here.

paths in the edited environment variables such as C:\Program Files\Git\bin become C:\Program Files\Giin where there is a \b backspace character in it.

This is interesting. Are you able to click on show environment contribution and see if WHO (for example Python extension specifically) is contributing to your corrupted environment?

Screenshot 2024-09-02 at 11 58 31 PM Screenshot 2024-09-02 at 11 58 39 PM

Also note that GitBash is currently not officially supported by Python extension at this moment: https://github.com/microsoft/vscode-python/issues/23382

If I opt out of this feature using the setting, then some of my other environments, which look slightly different for some reason, get even more corrupted such that basic commands like sed are not found and PATH is corrupted.

If this is the case, then the problem may reside outside of Python extension. What the Terminal Env Variable experiment does is, it tries to "activate" your environment using environment variables instead of explicitly running activate script on your terminal. You will see the "activate" line of command in terminal if you opt out of this experiment, but if that STILL causes a problem then it becomes very hard to tell because Python extension would not mingle around with your PATH if you opt-out of the experiment.

Are you able to "correctly" activate your environment outside of VS Code environment? (External terminal for example)

aatle commented 1 month ago

@anthonykim1 No worries, my workflow has not been inconvenienced by this. (Also, I did not know that GitBash is not officially supported currently, I use it as default on Windows. Hopefully it is officially supported soon.)

I am mostly certain that this bug is caused by the Microsoft Python extension, in certain conditions. The other shells I tested, powershell, cmd, GitBash outside of VS Code, they seemed to work fine when using poetry shell venv. Checking the Environment Contributors -> Show Environment Contributions, I see this file:

Environment Changes ```markdown # Terminal Environment Changes ## Extension: vscode.git Enables the following features: git auth provider - `GIT_ASKPASS=c:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass.sh` - `VSCODE_GIT_ASKPASS_NODE=C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\Code.exe` - `VSCODE_GIT_ASKPASS_EXTRA_ARGS=` - `VSCODE_GIT_ASKPASS_MAIN=c:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\resources\app\extensions\git\dist\askpass-main.js` - `VSCODE_GIT_IPC_HANDLE=\\.\pipe\vscode-git-b1ae1b85c2-sock` ## Extension: ms-python.python Activated environment for `~\AppData\Local\pypoetry\Cache\virtualenvs\non-package-mode-7fHQA28B-py3.12\Scripts\python.exe` - `EXEPATH=C:\Program Files\Git\bin` - `HOME=C:\Users\abcde` - `MSYS=disable_pcon` - `MSYSTEM=MINGW64` - `PATH=c:\Users\abcde\.vscode\extensions\ms-python.python-2024.12.3-win32-x64\python_files\deactivate\bash;C:\Users\abcde\AppData\Local\pypoetry\Cache\virtualenvs\non-package-mode-7fHQA28B-py3.12\Scripts;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Users\abcde\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0;C:\WINDOWS\System32\OpenSSH;C:\Program Files\Git\cmd;C:\Users\abcde\AppData\Local\Programs\Python\Python312\Scripts;C:\Users\abcde\AppData\Local\Programs\Python\Python312;C:\Users\abcde\AppData\Local\Microsoft\WindowsApps;C:\Users\abcde\AppData\Local\Programs\Microsoft VS Code\bin;C:\users\abcde\.local\bin;${env:PATH}` - `PLINK_PROTOCOL=ssh` - `PS1=(non-package-mode-py3.12) ${env:PS1}` - `PWD=C:/Users/abcde/AppData/Local/Programs/Microsoft VS Code` - `PYTHONIOENCODING=utf-8` - `PYTHONUNBUFFERED=1` - `TERM=xterm-256color` - `VIRTUAL_ENV=C:/Users/abcde/AppData/Local/pypoetry/Cache/virtualenvs/non-package-mode-7fHQA28B-py3.12` - `VIRTUAL_ENV_PROMPT=non-package-mode-py3.12` ``` This information and more is also available in the original post (some of it might be slightly outdated).

The invalid environment variables are: HOME, EXEPATH, and a part of PATH. The only other environment contributor is the git extension, and their variables seem fine. The issue does not occur when the Python extension is turned off.

I realized from more testing, it is indeed fixable without more issues by opting out of the terminal var experiment with settings.json. Turning off the feature makes the Python extension not contribute to the environment using variables, and instead uses activation script as you said.

"python.experiments.optOutFrom": ["pythonTerminalEnvVarActivation"],

Previously I said that this setting caused even worse errors on some projects, but it seems that this only applied to one specific older project I have, which I fixed. The issue will also only manifest if the selected python interpreter ("Python: Select Interpreter" command) is the poetry virtual environment one, not the global python interpreter, otherwise the venv doesn't get activated.

Conditions for bug, updated:

--> Causes certain Python extension environment variable contributions to be invalid because of backslashes in paths

github-actions[bot] commented 1 week ago

Hey @anthonykim1, this issue might need further attention.

@aatle, you can help us out by closing this issue if the problem no longer exists, or adding more information.