tox-dev / tox

Command line driven CI frontend and development task automation tool.
https://tox.wiki
MIT License
3.67k stars 520 forks source link

tox doesn't work as expected with Microsoft Store Pythons #2202

Closed tcztzy closed 3 years ago

tcztzy commented 3 years ago

I installed multiple version Python from Windows Store, their versions are:

  1. Python3.7 3.7.2544.0
  2. Python3.8 3.8.2800.0
  3. Python3.9 3.9.2032.0
  4. Python3.10 (RC) 3.10.193.0

When I use tox -vve py37 I got the error message at last of this issue.

As we can see in following codes, base_discover and locate_via_pep514 both failed, so the executable is resolved by py.path, and finally found the "C:\Users\\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe" which cannot be read directly.

https://github.com/tox-dev/tox/blob/156d4961dd50ee629c3c4c78c09ba6b1c0da76d3/src/tox/interpreters/windows/__init__.py#L12-L33

The problem is due to the limitation of Windows Store apps, Python's developer discussed about this, and then fix it in this.

Since [py](https://github.com/pytest-dev/py) is maintain only, I think this problem should be solved in tox side.

Thanks for your efforts!

VERBOSE MESSAGE

using tox.ini: C:\Users\Tang\Documents\pyproject-toml\pyproject.toml (pid 22660) removing C:\Users\Tang\Documents\pyproject-toml.tox\log using tox-3.24.3 from c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox__init__.py (pid 22660) c:\users\tang\documents\pyproject-toml.venv\scripts\python.exe (c:\users\tang\documents\pyproject-toml.venv\scripts\python.exe) is {'executable': 'c:\users\tang\documents\pyproject-toml\.venv\scripts\python.exe', 'implementation': 'CPython', 'version_info': [3, 10, 0, 'candidate', 1], 'version': '3.10.0rc1 (tags/v3.10.0rc1:cc115e5, Aug 3 2021, 15:22:01) [MSC v.1929 64 bit (AMD64)]', 'is_64': True, 'sysplatform': 'win32', 'os_sep': '\', 'extra_version_info': None} .package uses c:\users\tang\documents\pyproject-toml.venv\scripts\python.exe .package start: getenv C:\Users\Tang\Documents\pyproject-toml.tox.package .package reusing: C:\Users\Tang\Documents\pyproject-toml.tox.package .package finish: getenv C:\Users\Tang\Documents\pyproject-toml.tox.package after 0.15 seconds .package start: finishvenv .package finish: finishvenv after 0.00 seconds .package start: get-build-requires C:\Users\Tang\Documents\pyproject-toml.tox.package setting PATH=C:\Users\Tang\Documents\pyproject-toml.tox.package\Scripts;C:\Users\Tang\Documents\pyproject-toml.venv\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Tang\scoop\apps\nodejs\current\bin;C:\Users\Tang\scoop\apps\nodejs\current;C:\Users\Tang\AppData\Local\Pub\Cache\bin;C:\Users\Tang.cargo\bin;C:\Users\Tang\scoop\apps\yarn\current\Yarn\bin;C:\Users\Tang\scoop\apps\yarn\current\global\node_modules.bin;C:\Users\Tang\scoop\apps\android-sdk\current\tools\bin;C:\Users\Tang\Documents\flutter\bin;C:\Users\Tang\AppData\Local\Programs\Python\Python39\libs\;C:\Users\Tang\Documents\emsdk;C:\Users\Tang\Documents\emsdk\node\14.15.5_64bit\bin;C:\Users\Tang\Documents\emsdk\upstream\emscripten;C:\Users\Tang\scoop\shims;C:\Users\Tang\AppData\Local\Microsoft\WindowsApps;C:\Users\Tang\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Tang\AppData\Local\Programs\texlive\2021\bin\win32 [26260] C:\Users\Tang\Documents\pyproject-toml$ 'C:\Users\Tang\Documents\pyproject-toml.tox.package\Scripts\python' '.venv\Lib\site-packages\tox\helper\build_requires.py' pyproject_toml.build_system.build_backend '' . >.tox.package\log.package-6.log .package finish: get-build-requires C:\Users\Tang\Documents\pyproject-toml.tox.package after 0.50 seconds .package start: perform-isolated-build C:\Users\Tang\Documents\pyproject-toml.tox.package setting PATH=C:\Users\Tang\Documents\pyproject-toml.tox.package\Scripts;C:\Users\Tang\Documents\pyproject-toml.venv\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Tang\scoop\apps\nodejs\current\bin;C:\Users\Tang\scoop\apps\nodejs\current;C:\Users\Tang\AppData\Local\Pub\Cache\bin;C:\Users\Tang.cargo\bin;C:\Users\Tang\scoop\apps\yarn\current\Yarn\bin;C:\Users\Tang\scoop\apps\yarn\current\global\node_modules.bin;C:\Users\Tang\scoop\apps\android-sdk\current\tools\bin;C:\Users\Tang\Documents\flutter\bin;C:\Users\Tang\AppData\Local\Programs\Python\Python39\libs\;C:\Users\Tang\Documents\emsdk;C:\Users\Tang\Documents\emsdk\node\14.15.5_64bit\bin;C:\Users\Tang\Documents\emsdk\upstream\emscripten;C:\Users\Tang\scoop\shims;C:\Users\Tang\AppData\Local\Microsoft\WindowsApps;C:\Users\Tang\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Tang\AppData\Local\Programs\texlive\2021\bin\win32 [16224] C:\Users\Tang\Documents\pyproject-toml$ 'C:\Users\Tang\Documents\pyproject-toml.tox.package\Scripts\python' '.venv\Lib\site-packages\tox\helper\build_isolated.py' '.tox\dist' pyproject_toml.build_system.build_backend '' . >.tox.package\log.package-7.log running sdist running egg_info writing pyproject_toml.egg-info\PKG-INFO writing dependency_links to pyproject_toml.egg-info\dependency_links.txt writing requirements to pyproject_toml.egg-info\requires.txt writing top-level names to pyproject_toml.egg-info\top_level.txt reading manifest file 'pyproject_toml.egg-info\SOURCES.txt' writing manifest file 'pyproject_toml.egg-info\SOURCES.txt' running check warning: check: missing meta-data: either (author and author_email) or (maintainer and maintainer_email) should be supplied

creating pyproject-toml-0.0.11 creating pyproject-toml-0.0.11\pyproject_toml creating pyproject-toml-0.0.11\pyproject_toml.egg-info creating pyproject-toml-0.0.11\pyproject_toml\build_system creating pyproject-toml-0.0.11\pyproject_toml\tool copying files to pyproject-toml-0.0.11... copying README.rst -> pyproject-toml-0.0.11 copying pyproject.toml -> pyproject-toml-0.0.11 copying pyproject_toml__init__.py -> pyproject-toml-0.0.11\pyproject_toml copying pyproject_toml\project.py -> pyproject-toml-0.0.11\pyproject_toml copying pyproject_toml.egg-info\PKG-INFO -> pyproject-toml-0.0.11\pyproject_toml.egg-info copying pyproject_toml.egg-info\SOURCES.txt -> pyproject-toml-0.0.11\pyproject_toml.egg-info copying pyproject_toml.egg-info\dependency_links.txt -> pyproject-toml-0.0.11\pyproject_toml.egg-info copying pyproject_toml.egg-info\requires.txt -> pyproject-toml-0.0.11\pyproject_toml.egg-info copying pyproject_toml.egg-info\top_level.txt -> pyproject-toml-0.0.11\pyproject_toml.egg-info copying pyproject_toml\build_system__init.py -> pyproject-toml-0.0.11\pyproject_toml\build_system copying pyproject_toml\build_system\build_backend.py -> pyproject-toml-0.0.11\pyproject_toml\build_system copying pyproject_toml\tool\init__.py -> pyproject-toml-0.0.11\pyproject_toml\tool copying pyproject_toml\tool\pyproject_toml.py -> pyproject-toml-0.0.11\pyproject_toml\tool Writing pyproject-toml-0.0.11\setup.cfg Creating tar archive removing 'pyproject-toml-0.0.11' (and everything under it) pyproject-toml-0.0.11.tar.gz

.package finish: perform-isolated-build C:\Users\Tang\Documents\pyproject-toml.tox.package after 0.55 seconds copying new sdistfile to 'C:\Users\Tang\.tox\distshare\pyproject-toml-0.0.11.tar.gz' package .tmp\package\1\pyproject-toml-0.0.11.tar.gz links to dist\pyproject-toml-0.0.11.tar.gz (C:\Users\Tang\Documents\pyproject-toml.tox) python3.7 (C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\python3.7.EXE) is {'executable': 'C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe', 'implementation': 'CPython', 'version_info': [3, 7, 9, 'final', 0], 'version': '3.7.9 (tags/v3.7.9:13c94747c7, Aug 17 2020, 16:30:00) [MSC v.1900 64 bit (AMD64)]', 'is_64': True, 'sysplatform': 'win32', 'os_sep': '\', 'extra_version_info': None} py37 uses C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe py37 start: getenv C:\Users\Tang\Documents\pyproject-toml.tox\py37 py37 cannot reuse: no previous config C:\Users\Tang\Documents\pyproject-toml.tox\py37.tox-config1 py37 create: C:\Users\Tang\Documents\pyproject-toml.tox\py37 removing C:\Users\Tang\Documents\pyproject-toml.tox\py37 setting PATH=C:\Users\Tang\Documents\pyproject-toml.tox\py37\Scripts;C:\Users\Tang\Documents\pyproject-toml.venv\Scripts;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\Docker\Docker\resources\bin;C:\ProgramData\DockerDesktop\version-bin;C:\Users\Tang\scoop\apps\nodejs\current\bin;C:\Users\Tang\scoop\apps\nodejs\current;C:\Users\Tang\AppData\Local\Pub\Cache\bin;C:\Users\Tang.cargo\bin;C:\Users\Tang\scoop\apps\yarn\current\Yarn\bin;C:\Users\Tang\scoop\apps\yarn\current\global\node_modules.bin;C:\Users\Tang\scoop\apps\android-sdk\current\tools\bin;C:\Users\Tang\Documents\flutter\bin;C:\Users\Tang\AppData\Local\Programs\Python\Python39\libs\;C:\Users\Tang\Documents\emsdk;C:\Users\Tang\Documents\emsdk\node\14.15.5_64bit\bin;C:\Users\Tang\Documents\emsdk\upstream\emscripten;C:\Users\Tang\scoop\shims;C:\Users\Tang\AppData\Local\Microsoft\WindowsApps;C:\Users\Tang\AppData\Local\Programs\Microsoft VS Code\bin;C:\Users\Tang\AppData\Local\Programs\texlive\2021\bin\win32 [11216] C:\Users\Tang\Documents\pyproject-toml.tox$ 'c:\users\tang\documents\pyproject-toml.venv\scripts\python.exe' -m virtualenv --no-download --python 'C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe' py37 created virtual environment CPython3.7.9.final.0-64 in 1105ms creator Venv(dest=C:\Users\Tang\Documents\pyproject-toml.tox\py37, clear=False, no_vcs_ignore=False, global=False, describe=CPython3Windows) seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=C:\Users\Tang\AppData\Local\pypa\virtualenv) added seed packages: pip==21.2.3, setuptools==57.4.0, wheel==0.37.0 activators BashActivator,BatchActivator,FishActivator,PowerShellActivator,PythonActivator py37 finish: getenv C:\Users\Tang\Documents\pyproject-toml.tox\py37 after 1.87 seconds py37 start: installpkg C:\Users\Tang\Documents\pyproject-toml.tox.tmp\package\1\pyproject-toml-0.0.11.tar.gz py37 inst: C:\Users\Tang\Documents\pyproject-toml.tox.tmp\package\1\pyproject-toml-0.0.11.tar.gz ____ summary ____ py37: commands succeeded congratulations :) cleanup C:\Users\Tang\Documents\pyproject-toml.tox.tmp\package\1\pyproject-toml-0.0.11.tar.gz Traceback (most recent call last): File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\py_error.py", line 66, in checked_call return func(*args, **kwargs) OSError: [Errno 22] Invalid argument: 'C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe'

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.193.0_x64qbz5n2kfra8p0\lib\runpy.py", line 196, in _run_module_as_main return _run_code(code, main_globals, None, File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.10_3.10.193.0_x64qbz5n2kfra8p0\lib\runpy.py", line 86, in _run_code exec(code, run_globals) File "C:\Users\Tang\Documents\pyproject-toml.venv\Scripts\tox.exe__main.py", line 7, in File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session__init.py", line 44, in cmdline
main(args) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session\
init__.py", line 69, in main exit_code = session.runcommand() File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session\
init__.py", line 197, in runcommand return self.subcommand_test() File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session\
init__.py", line 225, in subcommand_test run_sequential(self.config, self.venv_dict) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session\commands\run\sequential.py", line 18, in run_sequential installpkg(venv, venv.package) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\session\commands\run\sequential.py", line 46, in installpkg venv.installpkg(path, action) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\venv.py", line 396, in installpkg self.install_pkg(sdistpath, action, "inst") File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\venv.py", line 376, in install_pkg self.finish() File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\venv.py", line 318, in finish live_config = self._getliveconfig() File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\venv.py", line 289, in _getliveconfig base_resolved_python_sha256 = getdigest(base_resolved_python_path) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\tox\venv.py", line 667, in getdigest return path.computehash("sha256") File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\py_path\local.py", line 240, in computehash
f = self.open('rb') File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\py_path\local.py", line 369, in open return py.error.checked_call(open, self.strpath, mode) File "c:\users\tang\documents\pyproject-toml.venv\lib\site-packages\py_error.py", line 86, in checked_call raise cls("%s%r" % (func.
name__, args)) py.error.ENOTDIR: [Not a directory]: open('C:\Users\Tang\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.7_qbz5n2kfra8p0\python.exe', 'rb')

tcztzy commented 3 years ago

I try to patch check_with_path like following, does it help? https://github.com/tox-dev/tox/blob/156d4961dd50ee629c3c4c78c09ba6b1c0da76d3/src/tox/interpreters/via_path.py#L17

_python_path_cache = {}

PATH_QUERY_SCRIPT = "import _winapi;print(_winapi.GetModuleFileName(0), end='')"

def get_python_path(cmd):
    try:
        return _python_path_cache[cmd].copy()
    except KeyError:
        pass
    proc = subprocess.Popen(
        [cmd, "-c", PATH_QUERY_SCRIPT],
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE,
        universal_newlines=True,
    )
    out, err = proc.communicate()
    if not proc.returncode:
        try:
            result = out
        except ValueError as exception:
            failure = exception
        else:
            _python_path_cache[cmd] = result
            return result.copy()
    else:
        failure = "exit code {}".format(proc.returncode)
    reporter.verbosity1("{!r} cmd {!r} out {!r} err {!r} ".format(failure, cmd, out, err))

def _patch(func):
    @wraps(func)
    def wrapper(*args, **kwds):
        result = func(*args, **kwds)
        if result is None or not result.lower().startswith(
            os.path.join(
                os.environ.get("LOCALAPPDATA", ""),
                "Microsoft",
                "WindowsApps",
                "PythonSoftwareFoundation.Python"
            ).lower()
        ):
            return result
        return get_python_path(result)
    return wrapper

check_with_path = _patch(_check_with_path)
gaborbernat commented 3 years ago

Can you try with https://pypi.org/project/tox/4.0.0a8/, that should fix it I think.

tcztzy commented 3 years ago

Can you try with https://pypi.org/project/tox/4.0.0a8/, that should fix it I think.

Thanks, it works now.