[Win] doesn't work with python from MS Store

[imba-tjd]

Have rebooted.

> userpath append C:\Users\xxx\.local\bin
The directory `C:\Users\xxx\.local\bin` is already in PATH, pending a shell restart! If you are sure you want to proceed, try again with the -f/--force flag.

> userpath append C:\Users\xxx\.local\bin -f
Success!

> ls C:\Users\xxx\.local\bin
iptest.exe  iptest3.exe  ipython.exe  ipython3.exe

> userpath verify C:\Users\xxx\.local\bin
The directory `C:\Users\xxx\.local\bin` is in PATH, pending a shell restart!

> ipython
'ipython' is not recognized as an internal or external command,
operable program or batch file.

> echo %path%
C:\Program Files\PowerShell\7;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\MacType;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\dotnet\;C:\Program Files\nodejs\;C:\Program Files\PowerShell\7;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Users\xxx\AppData\Local\Programs\Python\Launcher\;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\usr\bin;C:\Users\xxx\.dotnet\tools;C:\Users\xxx\AppData\Local\GitHubDesktop\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn;C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\amd64;C:\mingw64\bin;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Users\xxx\AppData\Roaming\npm;C:\Users\xxx\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\local-packages\Python38\Scripts;C:\tools;C:\Program Files\LLVM\bin;C:\Program Files\heroku\bin;C:\Users\xxx\.dotnet\tools;

[ofek]

Did you restart your shell?

[imba-tjd]

I have tested python from python.org and it works. So I think it's the limitation from MS Store. Don't know whether this could be fixed.

[ofek]

Can you please try restarting your shell?

[imba-tjd]

I had said that I have rebooted. If you insist, yes, I did restart my shell.

[ofek]

What shell are you using?

[imba-tjd]

cmd and powershell and pwsh. The 1st floor is cmd.

[ofek]

Ah! So you tried this with nested shells?

[imba-tjd]

no…… I test this with different shells.

Maybe the "1st floor" usage only exist in my native language. I actually mean in the first post I used cmd:

Well, now I find in the first post it is pwsh that I actually got the path result from. Only in this way can the PowerShell\7 be the top of the list. Anyway that doesn't matter too much.

[ofek]

Oh I see, thanks! Now that you've restarted can I see your PATH once more?

[imba-tjd]

> echo %path%
C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\MacType;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\dotnet\;C:\Program Files\nodejs\;C:\Program Files\PowerShell\7;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Users\xxx\AppData\Local\Programs\Python\Launcher\;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\usr\bin;C:\Users\xxx\.dotnet\tools;C:\Users\xxx\AppData\Local\GitHubDesktop\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn;C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\amd64;C:\mingw64\bin;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Users\xxx\AppData\Roaming\npm;C:\Users\xxx\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts;C:\tools;C:\Program Files\LLVM\bin;C:\Program Files\heroku\bin;C:\Users\xxx\.dotnet\tools;

[ofek]

Can you try:

powershell -Command "& {[Environment]::GetEnvironmentVariable('PATH', 'User')}"

[imba-tjd]

I removed some unused and duplicated entries. Of course userpath verify C:\Users\xxx\.local\bin still shows "The directory is in PATH".

Current:

Powershell:

PS C:\Users\xxx> & {[Environment]::GetEnvironmentVariable('PATH', 'User')}
C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\usr\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn;C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\amd64;C:\mingw64\bin;C:\Users\xxx\AppData\Roaming\npm;C:\Users\xxx\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts;C:\tools;C:\Program Files\LLVM\bin;C:\Program Files\heroku\bin;C:\Users\xxx\.dotnet\tools;

cmd:

> echo %path%
C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\MacType;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\dotnet\;C:\Program Files\nodejs\;C:\Program Files\PowerShell\7;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Users\xxx\AppData\Local\Microsoft\WindowsApps;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\mingw32\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\CommonExtensions\Microsoft\TeamFoundation\Team Explorer\Git\usr\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\Roslyn;C:\Users\xxx\AppData\Local\Programs\Microsoft VS Code\bin;C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\MSBuild\Current\Bin\amd64;C:\mingw64\bin;C:\Users\xxx\AppData\Roaming\npm;C:\Users\xxx\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\Scripts;C:\tools;C:\Program Files\LLVM\bin;C:\Program Files\heroku\bin;C:\Users\xxx\.dotnet\tools;

[ofek]

I'm extremely confused. I'll have to think about this.

[DataGhost]

I'm having the same issue, also with the Windows Store version. It did not even occur to me, but recent experiences with Windows Store apps (trying to modify them etc) lead me to some sources saying that the whole Store business executes in some kind of sandbox somehow. It does end up in the registry (which I can only assume is virtual, since searching with regedit did not yield any results) but does not appear to end up in any working environment.

Starting the python interpreter from the commandline seems to supply it with the environment of the commandline (logical) so it does not load the Environment key from the virtual registry. Querying, however, yields:

>>> with winreg.OpenKey(winreg.HKEY_CURRENT_USER, 'Environment', 0, winreg.KEY_READ) as key:
...     print(winreg.QueryValueEx(key, 'PATH'))
...
('%USERPROFILE%\\AppData\\Local\\Microsoft\\WindowsApps;;C:\\Users\\Username\\.local\\bin', 2)
>>> os.environ.get("PATH", "")
'C:\\WINDOWS\\system32;C:\\WINDOWS;C:\\WINDOWS\\System32\\Wbem;C:\\WINDOWS\\System32\\WindowsPowerShell\\v1.0\\;C:\\WINDOWS\\System32\\OpenSSH\\;C:\\Users\\Username\\AppData\\Local\\Microsoft\\WindowsApps;'

So userpath thinks all is correct but it'll never go live. I'm not sure this is fixable other than manually installing and updating Python, which kind of defeats the purpose of the store.

[ofek]

Found this: https://docs.python.org/3/using/windows.html#known-issues

Unfortunately, nothing can be done I think.

[DataGhost]

Well, I just hacked up an ugly workaround. Reads are from the normal registry, writes go to a private copy so that's when they start to go out of sync. It is possible to use subprocess to invoke something like reg add HKCU\Environment /v Path /t REG_EXPAND_SZ /d new_path. I just verified that this writes to the public registry.

[ofek]

@DataGhost If you come up with a way to detect Store Python I'll do this

[DataGhost]

>>> import sys
>>> sys.executable
'C:\\Users\\Username\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\\python.exe'

This is a user-specific path, but will always have the WindowsApps component. Something like this should work:

>>> def is_store():
...     import sys
...     return r"\AppData\Local\Microsoft\WindowsApps\" in sys.executable
...
>>> is_store()
True

and returns False on another machine where I have the non-store version. Although I'd say this could/should be fixed by modifying the winreg API to use these calls on Store versions but umm I doubt they're going to like the suggestion.

[ofek]

Thanks! One more question: does invoking userpath as subprocess work? It would be easier to call itself (with env var to prevent fork bomb)

[DataGhost]

I'm unsure what you mean or what you're trying to achieve by calling it in a subprocess. I'm not using userpath directly, I only saw that installing pipx and then running pipx ensurepath (which invoked userpath) didn't seem to have any effect. As long as the main executable lives in the WindowsApps directory, it seems to execute inside this sandbox-like thing. The def I just posted isn't flawless though. I just tried it in a virtualenv which has a stub binary in another location (...\venvname\Scripts\python.exe) and it does not write to the registry (using winreg) even though is_store() returned False.

Edit: replacing sys.executable with os.environ.get("PYTHONUSERBASE","") should fix the function, that also works in a venv. Edit2: sys.base_exec_prefix might be even better, that will contain something like C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.8_3.8.1008.0_x64__qbz5n2kfra8p0, even in a venv.

[ofek]

Thank you, I'll fix tomorrow

[Kurt-von-Laven]

Has this issue already been fixed, or no?

[ofek]

No, I haven't had time. Feel free to make a PR 🙂

[Kurt-von-Laven]

Sorry, but this issue would take me a disproportionate amount of time to fix relative to its value in light of my own ignorance of Windows. I am not sure if this is the right approach, but I am presently calling set "PATH=%PATH%;%USERPROFILE%\.local\bin" from a .bat script as a workaround for the fact that pipx ensurepath isn't working as intended.

[imba-tjd]

@ofek Hi. Do you consider to rewrite the logic to use setx.exe to modify PATH?

For example setx PATH "%PATH%;<newpath>" will permanently change PATH. Though I have no experience on checking.

[imba-tjd]

Well, I just tested again. Surprisingly it works. I don't know whether it's because now I'm using 3.10 or Win11.

[ofek]

Huh, maybe Win11?

[imba-tjd]

I tried 3.11b4 from Store on Win10 1903 and it works.