Windows embedable support

[karelv]

Issue

Installing virtualenv with pip install virtualenv should install the package.
I get error:

ModuleNotFoundError: No module named 'distlib'

Environment

Provide at least:

• OS: Win7 64bit

$ pip list
Package    Version
---------- -------
pip        20.0.2
setuptools 46.1.3
wheel      0.34.2

Python version: Python 3.8.2 (tags/v3.8.2:7b3ab59, Feb 25 2020, 23:03:10) [MSC v.1916 64 bit (AMD64)] on win32

Output of the virtual environment creation

Well output of installation process.

$ pip install virtualenv
Collecting virtualenv
  Using cached virtualenv-20.0.18-py2.py3-none-any.whl (4.6 MB)
Collecting appdirs<2,>=1.4.3
  Using cached appdirs-1.4.3-py2.py3-none-any.whl (12 kB)
Collecting distlib<1,>=0.3.0
  Using cached distlib-0.3.0.zip (571 kB)
    ERROR: Command errored out with exit status 1:
     command: 'c:\python382_64b\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\XXX\\AppData\\Local\\Temp\\pip-install-ms4c53bl\\distlib\\setup.py'"'"'; __file__='"'"'C:\\Users\\XXX\\AppData\\Local\\Temp\\pip-install-ms4c53bl\\distlib\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\XXX\AppData\Local\Temp\pip-install-ms4c53bl\distlib\pip-egg-info'
         cwd: C:\Users\XXX\AppData\Local\Temp\pip-install-ms4c53bl\distlib\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\XXX\AppData\Local\Temp\pip-install-ms4c53bl\distlib\setup.py", line 14, in <module>
        import distlib
    ModuleNotFoundError: No module named 'distlib'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

[karelv]

Earlier version seems to work

$ pip install "virtualenv<20"
Collecting virtualenv<20
  Downloading virtualenv-16.7.10-py2.py3-none-any.whl (3.4 MB)
     |████████████████████████████████| 3.4 MB 50 kB/s
Installing collected packages: virtualenv
Successfully installed virtualenv-16.7.10

However it seems not to be able to create a virtualenv.

$ virtualenv.exe pyxxx
Using base prefix 'c:\\python382_32b'
Traceback (most recent call last):
  File "runpy.py", line 193, in _run_module_as_main
  File "runpy.py", line 86, in _run_code
  File "C:\python382_32b\Scripts\virtualenv.exe\__main__.py", line 7, in <module>
  File "c:\python382_32b\lib\site-packages\virtualenv.py", line 859, in main
    create_environment(
  File "c:\python382_32b\lib\site-packages\virtualenv.py", line 1161, in create_environment
    install_python(home_dir, lib_dir, inc_dir, bin_dir, site_packages=site_packages, clear=clear, symlink=symlink)
  File "c:\python382_32b\lib\site-packages\virtualenv.py", line 1467, in install_python
    writefile(site_filename_dst, SITE_PY)
  File "c:\python382_32b\lib\site-packages\virtualenv.py", line 431, in writefile
    with open(dest, "wb") as f:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\pyxxx\\python38.zip\\site.py'

[pfmoore]

I can't reproduce this. My versions of pip, setuptools and wheel are the same, this is in a fresh Python 3.8.0 venv. I am using Windows 10, but I'm not sure how that would make a difference here.

Your prompt is $, which is not usual for the standard Windows shells (cmd or powershell). Are you using an unusual environment such as cygwin? That's the only thing I can think of which could explain the difference with your results and mine.

[gaborbernat]

This feels like a pip issue; as it's related to installing the package. For now, we still need a reproduced of this. Can you provide more detailed information?

[pfmoore]

I just noticed:

FileNotFoundError: [Errno 2] No such file or directory: 'C:\\pyxxx\\python38.zip\\site.py'

It looks like you have the stdlib in a zip file. That's unusual - are you by any chance trying to use the embedded Python distribution? That's not intended for "general" use, and could be why you're getting issues here.

Please describe how you installed your Python environment, as it may be relevant.

[gaborbernat]

Closing for now as we've got no response to our request for more information. If you can acquire those please post the details and we'll reopen the issue.

[karelv]

It is indeed the embedded python version. My goal was to be try the new python version along my previous version, it appeared to me like the most obvious thing to do, however it seems not so obvious... Thanks for having a look at this, so quickly! (I will have to find another way for trying new python version)

[gaborbernat]

How did you install an embedded Python?

[pfmoore]

@gaborbernat I was specifically talking about the windows embeddable package, which is available as a zipfile that can be unzipped anywhere.

The problem is that while it appears to work like "normal" Python, it's tuned specifically for embedding into your own application - so, for example, it doesn't include pip, and you need to do some config changes to get pip to work. It's not really a supported use of that distribution.

For virtualenv, I guess there is a question of whether you want to support setups where the standard library is zipped up and/or contains only .pyc files (both of which are true for the embedded distribution).

[gaborbernat]

Working only with pyc files is already supported. Them being zipped up is an interesting question; which boils down to the fact can we even support these without unzipping the entire library.

[karelv]

I unzipped the embedded package: https://www.python.org/ftp/python/3.8.2/python-3.8.2-embed-win32.zip Then I run the get-pip.py: https://bootstrap.pypa.io/get-pip.py Then I uncomment in python38._pth the import statement. Then I add some path to PATH env variable. (add path to python.exe and pip.exe) Then I run pip install virtualenv.

[gaborbernat]

Wow that feels complicated for a newcomer. Unless there's a more just install/extract and use way of this I'm tempted that to say that this is not supported.

[karelv]

Up to you. I found that pretty straight forward. Especially if you want to create a python environment without installation.

I'm ok with your either your decision, as I somehow have my virtual environment using my above steps. (I can have as many copies of that as I want...)

[arjd]

I ran into the same issue and was not able to get virtualenv working with karelv's workaround. My use-case was creating a fully-functional portable Python installation, without having to use anaconda.

[jcrben]

My use case is the same as above - wanting something portable on Windows without dealing with registry changes. Embedded python doesn't ship with venv so I was hoping to use this.

I was able to run pip install virtualenv successfully and then create a venv running virtualenv ./venv. But after I activate it and run python -m pip list it returns only the non-virtualenv packages. I confirmed that which python and which pip are indeed running the virtualenv pip - and even invoked them directly.

Interestingly, pip install cowsay is installing into the virtualenv site-packages. pip list --local lists nothing tho. I then tried a simple test-cowsay.py and it wasn't able to import cowsay:

import cowsay

cowsay.cow('Hello World')

(venv) bncrsy@~/test1$ python test-python.py
Traceback (most recent call last):
  File "C:\Users\ben\test1\test-python.py", line 1, in <module>
    import cowsay
ModuleNotFoundError: No module named 'cowsay'

I was originally using "git bash" (which uses msys2) but I tried it in Powershell and got the the same thing.

I also tried dropping a python311._pth into the venv folder with this but I'm not really sure what I'm doing there.

.\Lib
.\python311
.\Scripts
.

# import site is consistent with non-embedded
# import site
# 

import site

I also saw https://github.com/pypa/pip/issues/9767

[pfmoore]

My use case is the same as above - wanting something portable on Windows without dealing with registry changes.

Thye nuget packages described here might be more suitable for that use case.

[jcrben]

Thanks @pfmoore - introducing a dependency on nuget (which I've never used before, Windows is definitely of a secondary thing for me) when it's not in my regular stack isn't ideal for my particular situation. One reason I'm avoiding the registry is that after getting a bit too fancy with scripting the regular installer some registry stuff got a bit screwy and spent an hour or two just deleting registry entries.

I ended up fixing it - python311._pth needed to be moved into the Scripts folder next to the python.exe executable and I added a line to the Lib folder of the original portable python so my revised python311._pth looks like this:

.\Lib
.\python311
.\Scripts
.
C:\Users\ben\bin\python-portable\Lib

# import site is consistent with non-embedded
# import site
# 

import site

It seems like this could be an easy fix - that path I added ( C:\Users\ben\bin\python-portable\Lib) lines up with the base-prefix already added to the pyvenv.cfg

pip list now properly reports just cowsay and running my test-cowsay.py also works