search

pypa / virtualenv

Virtual Python Environment builder
https://virtualenv.pypa.io
MIT License
4.82k stars 1.03k forks source link

framework build CPython macOs issues #1561

Closed rpkilby closed 4 years ago

rpkilby commented 4 years ago

Hi. I'm trying to create a Python 2 virtualenv on macOS Mojave and am running into an error.

bagel$ pip3 install --user virtualenv
  <snipped>
  WARNING: The script virtualenv is installed in '/Users/bagel/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed virtualenv-20.0.1

bagel$ pip3 list
Package            Version
------------------ -------
appdirs            1.4.3  
filelock           3.0.12 
importlib-metadata 1.5.0  
pip                19.3.1 
setuptools         42.0.2 
six                1.14.0 
virtualenv         20.0.1 
wheel              0.33.6 
zipp               2.2.0  

bagel$ which python2.7
/usr/bin/python2.7

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv
PermissionError: [Errno 1] Operation not permitted: '/Users/bagel/Documents/projects/test/venv/bin/python'

It's worth noting that Python 2 is the system Python, while Python 3 is installed from Homebrew.

gaborbernat commented 4 years ago

Can you please show the output with -vvv? Thanks!

rpkilby commented 4 years ago 
bagel$ ~/Library/Python/3.7/bin/virtualenv -vvv -p python2.7 venv
85 setup logging to NOTSET [DEBUG report:43]
97 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:44]
97 proposed PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.6 (default, Dec 30 2019, 19:38:28) \n[Clang 11.0.0 (clang-1100.0.33.16)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
97 discover PATH[0]=/usr/local/bin [DEBUG builtin:77]
98 filesystem is not case-sensitive [DEBUG info:28]
99 discover PATH[1]=/usr/local/sbin [DEBUG builtin:77]
99 discover PATH[2]=/usr/local/bin [DEBUG builtin:77]
99 discover PATH[3]=/usr/bin [DEBUG builtin:77]
100 Attempting to acquire lock 4505262672 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
100 Lock 4505262672 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
100 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:73]
101 Attempting to release lock 4505262672 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
101 Lock 4505262672 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
102 Attempting to acquire lock 4505327312 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:270]
102 Lock 4505327312 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:274]
102 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.json for /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG cached_py_info:73]
103 Attempting to release lock 4505327312 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:315]
104 Lock 4505327312 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.1/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:318]
104 proposed PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:50]
104 accepted PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:52]
131 create virtual environment via CPython2Posix(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False) [INFO session:24]
131 create folder /Users/bagel/Documents/projects/test/venv/bin [DEBUG _sync:22]
131 create folder /Users/bagel/Documents/projects/test/venv/include [DEBUG _sync:22]
132 create folder /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages [DEBUG _sync:22]
132 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
132     home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132     implementation = CPython [DEBUG pyenv_cfg:38]
132     version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
132     virtualenv = 20.0.1 [DEBUG pyenv_cfg:38]
132     include-system-site-packages = false [DEBUG pyenv_cfg:38]
132     base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132     base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
132     base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
132 copy /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python to /Users/bagel/Documents/projects/test/venv/bin/python [DEBUG _sync:49]
134 done in 132ms [INFO __main__:23]
134 PermissionError: [Errno 1] Operation not permitted: '/Users/bagel/Documents/projects/test/venv/bin/python' [ERROR __main__:30]

Also: 

Sherpa-3:test bagel$ which python2.7
/usr/bin/python2.7
Sherpa-3:test bagel$ ls /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Nov 16  2018 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
szmark001 commented 4 years ago

I am seeing the same issue with version 20.0.1.

mctrafik commented 4 years ago

+1 on Mac OS 10.15.3 Catalina. Same exact issue.

gaborbernat commented 4 years ago

So basically we do not support with the rewrite the MacOs framework python. We'll need to fix this. In this particular case it seems that copying the file stats of the system python is not allowed, hence the above error.

gaborbernat commented 4 years ago

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.2/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-2-2020-02-11) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

rpkilby commented 4 years ago

I don't think this is fixed entirely. Virtualenv creation doesn't fail, but there are a lot of unexpected packages present. Also, is the venv creation output expected?

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv 
created virtual environment CPython2Posix(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1 via=copy

bagel$ source venv/bin/activate

(venv) bagel$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                                Version 
-------------------------------------- --------
altgraph                               0.10.2  
bdist-mpkg                             0.5.0   
bonjour-py                             0.3     
macholib                               1.5.1   
matplotlib                             1.3.1   
modulegraph                            0.10.4  
numpy                                  1.8.0rc1
pip                                    20.0.2  
py2app                                 0.7.3   
pyobjc-core                            2.5.1   
pyobjc-framework-Accounts              2.5.1   
pyobjc-framework-AddressBook           2.5.1   
pyobjc-framework-AppleScriptKit        2.5.1   
pyobjc-framework-AppleScriptObjC       2.5.1   
pyobjc-framework-Automator             2.5.1   
pyobjc-framework-CFNetwork             2.5.1   
pyobjc-framework-Cocoa                 2.5.1   
pyobjc-framework-Collaboration         2.5.1   
pyobjc-framework-CoreData              2.5.1   
pyobjc-framework-CoreLocation          2.5.1   
pyobjc-framework-CoreText              2.5.1   
pyobjc-framework-DictionaryServices    2.5.1   
pyobjc-framework-EventKit              2.5.1   
pyobjc-framework-ExceptionHandling     2.5.1   
pyobjc-framework-FSEvents              2.5.1   
pyobjc-framework-InputMethodKit        2.5.1   
pyobjc-framework-InstallerPlugins      2.5.1   
pyobjc-framework-InstantMessage        2.5.1   
pyobjc-framework-LatentSemanticMapping 2.5.1   
pyobjc-framework-LaunchServices        2.5.1   
pyobjc-framework-Message               2.5.1   
pyobjc-framework-OpenDirectory         2.5.1   
pyobjc-framework-PreferencePanes       2.5.1   
pyobjc-framework-PubSub                2.5.1   
pyobjc-framework-QTKit                 2.5.1   
pyobjc-framework-Quartz                2.5.1   
pyobjc-framework-ScreenSaver           2.5.1   
pyobjc-framework-ScriptingBridge       2.5.1   
pyobjc-framework-SearchKit             2.5.1   
pyobjc-framework-ServiceManagement     2.5.1   
pyobjc-framework-Social                2.5.1   
pyobjc-framework-SyncServices          2.5.1   
pyobjc-framework-SystemConfiguration   2.5.1   
pyobjc-framework-WebKit                2.5.1   
pyOpenSSL                              0.13.1  
pyparsing                              2.0.1   
python-dateutil                        1.5     
pytz                                   2013.7  
scipy                                  0.13.0b1
setuptools                             44.0.0  
six                                    1.4.1   
vboxapi                                1.0     
wheel                                  0.34.2  
xattr                                  0.6.4

However, the contents of the venv's site-packages are:

bagel$ ls venv/lib/python2.7/site-packages/
total 4.0K
drwxr-xr-x  7 bagel staff  224 Feb 11 10:42 pip
drwxr-xr-x  9 bagel staff  288 Feb 11 10:41 pip-20.0.2.dist-info
drwxr-xr-x  6 bagel staff  192 Feb 11 10:41 pkg_resources
drwxr-xr-x 43 bagel staff 1.4K Feb 11 10:41 setuptools
drwxr-xr-x 11 bagel staff  352 Feb 11 10:41 setuptools-44.0.0.dist-info
drwxr-xr-x 13 bagel staff  416 Feb 11 10:41 wheel
drwxr-xr-x  9 bagel staff  288 Feb 11 10:41 wheel-0.34.2.dist-info
-rw-r--r--  1 bagel staff  126 Feb 11 10:41 easy_install.py
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 pip-20.0.2.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 setuptools-44.0.0.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 11 10:41 wheel-0.34.2.dist-info.virtualenv
gaborbernat commented 4 years ago

Thanks I'll take a look why this happens 🤔 seems the system python does not respects ignore global site packages on site.py 🤔

gaborbernat commented 4 years ago

https://github.com/pypa/virtualenv/pull/1588 fixes this now.

gaborbernat commented 4 years ago

Hello, a fix for this issue has been released via virtualenv 20.0.2; see https://pypi.org/project/virtualenv/20.0.3/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-3-2020-02-12) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

YannickJadoul commented 4 years ago

Something's still going wrong with Python 2 on macOS. As far as I notice, scripts are still installed globally. Is this possible?

This is output from Python 3 and Python 2, installed with HomeBrew.

Python 3, correctly installing the cibuildwheel script in the virtualenv's bin:

$ python3 -m virtualenv --version
virtualenv 20.0.3 from /usr/local/lib/python3.7/site-packages/virtualenv/__init__.py
$ python3 -m virtualenv TMPENV3
created virtual environment in 182ms CPython3Posix(dest=/Users/yannick/TMPENV3, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
$ . TMPENV3/bin/activate
(TMPENV3) $ pip install cibuildwheel
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Processing ./Library/Caches/pip/wheels/53/49/86/a05c97f04697533da8e8d0e7e4a88cfeeb8e5aea5616a65e4a/bashlex-0.14-py3-none-any.whl
Installing collected packages: bashlex, cibuildwheel
Successfully installed bashlex-0.14 cibuildwheel-1.1.0
(TMPENV3) $ which cibuildwheel
/Users/yannick/TMPENV3/bin/cibuildwheel
(TMPENV3) $ deactivate

Python 2, installing the script globally, it seems:

$ python -m virtualenv --version
virtualenv 20.0.3 from /usr/local/lib/python2.7/site-packages/virtualenv/__init__.pyc
$ python -m virtualenv TMPENV
created virtual environment in 183ms CPython2Posix(dest=/Users/yannick/TMPENV, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
$ . TMPENV/bin/activate
(TMPENV) $ which pip
/Users/yannick/TMPENV/bin/pip
(TMPENV) $ pip install cibuildwheel
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: bashlex!=0.13 in /usr/local/lib/python2.7/site-packages (from cibuildwheel) (0.14)
Requirement already satisfied: enum34 in /usr/local/lib/python2.7/site-packages (from bashlex!=0.13->cibuildwheel) (1.1.6)
Installing collected packages: cibuildwheel
Successfully installed cibuildwheel-1.1.0
(TMPENV) $ which cibuildwheel
/usr/local/bin/cibuildwheel
gaborbernat commented 4 years ago

Does it install globally even if you don't use the activate, but invoke pip directly?

YannickJadoul commented 4 years ago

Does it install globally even if you don't use the activate, but invoke pip directly?

Like this, you mean? Then yes:

ai31:~ yannick$ python -m virtualenv TMPENV_BIS
created virtual environment in 184ms CPython2Posix(dest=/Users/yannick/TMPENV_BIS, clear=False, global=False) with seeder FromAppData pip=latest setuptools=latest wheel=latest app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1 via=copy
ai31:~ yannick$ TMPENV_BIS/bin/pip install cibuildwheel
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Collecting cibuildwheel
  Using cached cibuildwheel-1.1.0-py2.py3-none-any.whl (24 kB)
Requirement already satisfied: bashlex!=0.13 in /usr/local/lib/python2.7/site-packages (from cibuildwheel) (0.14)
Requirement already satisfied: enum34 in /usr/local/lib/python2.7/site-packages (from bashlex!=0.13->cibuildwheel) (1.1.6)
Installing collected packages: cibuildwheel
Successfully installed cibuildwheel-1.1.0
ai31:~ yannick$ which cibuildwheel
/usr/local/bin/cibuildwheel
gaborbernat commented 4 years ago

Will check what's happening, thanks!

YannickJadoul commented 4 years ago

Will check what's happening, thanks!

Thank yóu for the quick follow-up. If you still need anything from me, do let me know!

rpkilby commented 4 years ago

Just throwing this out there - I wonder if this is somehow related to #1095.

ssbarnea commented 4 years ago

Is this bug the same reason for seeing https://dev.azure.com/pypa/virtualenv/_build/results?buildId=19042&view=logs&jobId=cc0564e1-d822-5f24-0bbb-f60430a68e65&j=cc0564e1-d822-5f24-0bbb-f60430a68e65&t=9fc317a6-b66c-593b-646c-b1040cd24f7e ?

gaborbernat commented 4 years ago

It could be, though should not.

gaborbernat commented 4 years ago

Hello, a fix for this issue has been released via virtualenv 20.0.5; see https://pypi.org/project/virtualenv/20.0.5/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-5-2020-02-21) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release.

thanks

YannickJadoul commented 4 years ago

Thank you, @gaborbernat!

I'm trying out the new release at joerick/cibuildwheel#274, right now, and half of our problem is already gone (see first commit, succeedig). But something's still seems to be happening with the CircleCI build, running our tests in a virtualenv (see second commit, failing) :-/ I'll try to look further into it and see if I can make a minimal example.

gaborbernat commented 4 years ago

@YannickJadoul can't seem to understand what's happening in there, seems pypy and pypy3 is mixed in together making it hard to see what's happening. If you can create a minimal report that would be great.

rpkilby commented 4 years ago

Hi @gaborbernat. I've upgraded to virtualenv 20.0.5, but am still seeing this issue: https://github.com/pypa/virtualenv/issues/1561#issuecomment-584793979

gaborbernat commented 4 years ago

Can you post a fresh trace exposing the issue?

rpkilby commented 4 years ago 
bagel$ ~/Library/Python/3.7/bin/virtualenv --version
virtualenv 20.0.5 from /Users/bagel/Library/Python/3.7/lib/python/site-packages/virtualenv/__init__.py

bagel$ ~/Library/Python/3.7/bin/virtualenv -p python2.7 venv
created virtual environment CPython2.7.10.final.0-64 in 235ms
  creator CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator

bagel$ source venv/bin/activate

bagel$ pip list
DEPRECATION: Python 2.7 reached the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 is no longer maintained. A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support
Package                                Version 
-------------------------------------- --------
altgraph                               0.10.2  
bdist-mpkg                             0.5.0   
bonjour-py                             0.3     
macholib                               1.5.1   
matplotlib                             1.3.1   
modulegraph                            0.10.4  
numpy                                  1.8.0rc1
pip                                    20.0.2  
py2app                                 0.7.3   
pyobjc-core                            2.5.1   
pyobjc-framework-Accounts              2.5.1   
pyobjc-framework-AddressBook           2.5.1   
pyobjc-framework-AppleScriptKit        2.5.1   
pyobjc-framework-AppleScriptObjC       2.5.1   
pyobjc-framework-Automator             2.5.1   
pyobjc-framework-CFNetwork             2.5.1   
pyobjc-framework-Cocoa                 2.5.1   
pyobjc-framework-Collaboration         2.5.1   
pyobjc-framework-CoreData              2.5.1   
pyobjc-framework-CoreLocation          2.5.1   
pyobjc-framework-CoreText              2.5.1   
pyobjc-framework-DictionaryServices    2.5.1   
pyobjc-framework-EventKit              2.5.1   
pyobjc-framework-ExceptionHandling     2.5.1   
pyobjc-framework-FSEvents              2.5.1   
pyobjc-framework-InputMethodKit        2.5.1   
pyobjc-framework-InstallerPlugins      2.5.1   
pyobjc-framework-InstantMessage        2.5.1   
pyobjc-framework-LatentSemanticMapping 2.5.1   
pyobjc-framework-LaunchServices        2.5.1   
pyobjc-framework-Message               2.5.1   
pyobjc-framework-OpenDirectory         2.5.1   
pyobjc-framework-PreferencePanes       2.5.1   
pyobjc-framework-PubSub                2.5.1   
pyobjc-framework-QTKit                 2.5.1   
pyobjc-framework-Quartz                2.5.1   
pyobjc-framework-ScreenSaver           2.5.1   
pyobjc-framework-ScriptingBridge       2.5.1   
pyobjc-framework-SearchKit             2.5.1   
pyobjc-framework-ServiceManagement     2.5.1   
pyobjc-framework-Social                2.5.1   
pyobjc-framework-SyncServices          2.5.1   
pyobjc-framework-SystemConfiguration   2.5.1   
pyobjc-framework-WebKit                2.5.1   
pyOpenSSL                              0.13.1  
pyparsing                              2.0.1   
python-dateutil                        1.5     
pytz                                   2013.7  
scipy                                  0.13.0b1
setuptools                             44.0.0  
six                                    1.4.1   
vboxapi                                1.0     
wheel                                  0.34.2  
xattr                                  0.6.4
rpkilby commented 4 years ago

And for completeness

bagel$ ls venv/lib/python2.7/site-packages/
total 4.0K
drwxr-xr-x  7 bagel staff  224 Feb 21 11:33 pip
drwxr-xr-x  9 bagel staff  288 Feb 21 11:32 pip-20.0.2.dist-info
drwxr-xr-x  8 bagel staff  256 Feb 21 11:33 pkg_resources
drwxr-xr-x 56 bagel staff 1.8K Feb 21 11:33 setuptools
drwxr-xr-x 11 bagel staff  352 Feb 21 11:32 setuptools-44.0.0.dist-info
drwxr-xr-x 13 bagel staff  416 Feb 21 11:32 wheel
drwxr-xr-x  9 bagel staff  288 Feb 21 11:32 wheel-0.34.2.dist-info
-rw-r--r--  1 bagel staff  126 Feb 21 11:32 easy_install.py
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 pip-20.0.2.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 setuptools-44.0.0.dist-info.virtualenv
-rw-r--r--  1 bagel staff    0 Feb 21 11:32 wheel-0.34.2.dist-info.virtualenv
gaborbernat commented 4 years ago

The usual caveats apply add v three times to the creation so we can see what happened 👍and ideally with --clear.

gaborbernat commented 4 years ago

Also in your case would help if you can display the -m site of virtual environment 👍 and the global python you're creating the venv from.

rpkilby commented 4 years ago

Note that I installed virtualenv with pip3 install --user virtualenv but am creating a Python 2 virtualenv.

bagel$ ~/Library/Python/3.7/bin/virtualenv -vvv -p python2.7 venv --clear
55 setup logging to NOTSET [DEBUG report:43]
61 find interpreter for spec PythonSpec(implementation=CPython, major=2, minor=7) [INFO builtin:43]
61 proposed PythonInfo(spec=CPython3.7.6.final.0-64, exe=/usr/local/opt/python/bin/python3.7, platform=darwin, version='3.7.6 (default, Dec 30 2019, 19:38:28) \n[Clang 11.0.0 (clang-1100.0.33.16)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
61 discover PATH[0]=/usr/local/bin [DEBUG builtin:76]
63 filesystem is not case-sensitive [DEBUG info:28]
63 discover PATH[1]=/usr/local/sbin [DEBUG builtin:76]
63 discover PATH[2]=/usr/local/bin [DEBUG builtin:76]
63 discover PATH[3]=/usr/bin [DEBUG builtin:76]
64 Attempting to acquire lock 4530209296 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:270]
64 Lock 4530209296 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:274]
64 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.json for /usr/bin/python2.7 [DEBUG cached_py_info:72]
65 Attempting to release lock 4530209296 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [DEBUG filelock:315]
65 Lock 4530209296 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/beb4f305451b4835b36a63cb1c75e625a7251cf8d76b65be2f5856eee68cb551.lock [INFO filelock:318]
65 Attempting to acquire lock 4530325136 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:270]
65 Lock 4530325136 acquired on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:274]
65 get PythonInfo from /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.json for /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG cached_py_info:72]
66 Attempting to release lock 4530325136 on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [DEBUG filelock:315]
66 Lock 4530325136 released on /Users/bagel/Library/Application Support/virtualenv/py-info/20.0.5/545e036c4b32438aced1f6b3c8d38ca151d9c36189e05839cb0aa568fda70ddd.lock [INFO filelock:318]
66 proposed PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:49]
66 accepted PathPythonInfo(spec=CPython2.7.10.final.0-64, system=/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python, exe=/usr/bin/python2.7, platform=darwin, version='2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:51]
79 create virtual environment via CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=True, global=False) [INFO session:51]
80 create folder /Users/bagel/Documents/projects/test/venv/bin [DEBUG _sync:20]
80 create folder /Users/bagel/Documents/projects/test/venv/include [DEBUG _sync:20]
80 create folder /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages [DEBUG _sync:20]
80 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
80  home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80  implementation = CPython [DEBUG pyenv_cfg:38]
80  version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
80  virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
80  include-system-site-packages = false [DEBUG pyenv_cfg:38]
80  base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80  base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
80  base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
81 copy /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python to /Users/bagel/Documents/projects/test/venv/bin/python [DEBUG _sync:47]
81 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.py to /Users/bagel/Documents/projects/test/venv/lib/python2.7/os.py [DEBUG _sync:39]
81 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc to /Users/bagel/Documents/projects/test/venv/lib/python2.7/os.pyc [DEBUG _sync:39]
82 symlink directory /System/Library/Frameworks/Python.framework/Versions/2.7/include/python2.7 to /Users/bagel/Documents/projects/test/venv/include/python2.7 [DEBUG _sync:39]
82 symlink directory /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload to /Users/bagel/Documents/projects/test/venv/lib/python2.7/lib-dynload [DEBUG _sync:39]
82 symlink /System/Library/Frameworks/Python.framework/Versions/2.7/Python to /Users/bagel/Documents/projects/test/venv/.Python [DEBUG _sync:39]
82 change Mach-O for /Users/bagel/Documents/projects/test/venv/bin/python from /System/Library/Frameworks/Python.framework/Versions/2.7/Python to @executable_path/../.Python [DEBUG mac_os:77]
83 ============================== target debug ============================== [DEBUG session:53]
83 debug via /Users/bagel/Documents/projects/test/venv/bin/python /Users/bagel/Library/Python/3.7/lib/python/site-packages/virtualenv/create/debug.py [DEBUG creator:191]
83 {
  "sys": {
    "executable": "/Users/bagel/Documents/projects/test/venv/bin/python",
    "_base_executable": null,
    "prefix": "/Users/bagel/Documents/projects/test/venv",
    "base_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "real_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "exec_prefix": "/Users/bagel/Documents/projects/test/venv",
    "base_exec_prefix": "/System/Library/Frameworks/Python.framework/Versions/2.7",
    "path": [
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload",
      "/Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages",
      "/Library/Python/2.7/site-packages",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python",
      "/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC"
    ],
    "meta_path": [],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "2.7.10 (default, Feb 22 2019, 21:55:15) \n[GCC 4.2.1 Compatible Apple LLVM 10.0.1 (clang-1001.0.37.14)]",
  "os": "<module 'os' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/os.pyc'>",
  "site": "<module 'site' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.pyc'>",
  "datetime": "<module 'datetime' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/datetime.so'>",
  "math": "<module 'math' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so'>",
  "json": "<module 'json' from '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.pyc'>"
} [DEBUG session:54]
215 add seed packages via FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1) [INFO session:58]
215 Attempting to acquire lock 4531077264 on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:270]
216 Lock 4531077264 acquired on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:274]
217 install pip from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/pip-20.0.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
217 install setuptools from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/setuptools-44.0.0-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
218 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip-20.0.2.dist-info.virtualenv [DEBUG _sync:47]
218 install wheel from wheel /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels/wheel-0.34.2-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:56]
218 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/easy_install.py to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/easy_install.py [DEBUG _sync:47]
218 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel [DEBUG _sync:47]
219 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip [DEBUG _sync:47]
219 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools-44.0.0.dist-info.virtualenv [DEBUG _sync:47]
220 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools [DEBUG _sync:47]
230 copy /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info.virtualenv to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel-0.34.2.dist-info.virtualenv [DEBUG _sync:47]
231 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/wheel-0.34.2-py2.py3-none-any/wheel-0.34.2.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/wheel-0.34.2.dist-info [DEBUG _sync:47]
292 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel to 755 [INFO util:566]
293 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel3 to 755 [INFO util:566]
293 changing mode of /Users/bagel/Documents/projects/test/venv/bin/wheel-3.7 to 755 [INFO util:566]
293 generated console scripts wheel wheel-3.7 wheel3 [DEBUG base:48]
321 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/pkg_resources to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pkg_resources [DEBUG _sync:47]
332 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/setuptools-44.0.0-py2.py3-none-any/setuptools-44.0.0.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/setuptools-44.0.0.dist-info [DEBUG _sync:47]
338 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install3 to 755 [INFO util:566]
338 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install-3.7 to 755 [INFO util:566]
339 changing mode of /Users/bagel/Documents/projects/test/venv/bin/easy_install to 755 [INFO util:566]
339 generated console scripts easy_install-3.7 easy_install easy_install3 [DEBUG base:48]
441 copy directory /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/image/CopyPipInstall/pip-20.0.2-py2.py3-none-any/pip-20.0.2.dist-info to /Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages/pip-20.0.2.dist-info [DEBUG _sync:47]
445 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip to 755 [INFO util:566]
445 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip-3.7 to 755 [INFO util:566]
446 changing mode of /Users/bagel/Documents/projects/test/venv/bin/pip3 to 755 [INFO util:566]
446 generated console scripts pip-3.7 pip3 pip [DEBUG base:48]
446 Attempting to release lock 4531077264 on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [DEBUG filelock:315]
446 Lock 4531077264 released on /Users/bagel/Library/Application Support/virtualenv/seed-v1/2.7/wheels.lock [INFO filelock:318]
446 add activators for Bash, CShell, Fish, PowerShell, Python [INFO session:64]
449 write /Users/bagel/Documents/projects/test/venv/pyvenv.cfg [DEBUG pyenv_cfg:34]
449     home = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
449     implementation = CPython [DEBUG pyenv_cfg:38]
450     version_info = 2.7.10.final.0 [DEBUG pyenv_cfg:38]
450     virtualenv = 20.0.5 [DEBUG pyenv_cfg:38]
450     include-system-site-packages = false [DEBUG pyenv_cfg:38]
450     base-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
450     base-exec-prefix = /System/Library/Frameworks/Python.framework/Versions/2.7 [DEBUG pyenv_cfg:38]
450     base-executable = /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python [DEBUG pyenv_cfg:38]
450 created virtual environment CPython2.7.10.final.0-64 in 395ms
  creator CPython2macOsFramework(dest=/Users/bagel/Documents/projects/test/venv, clear=True, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/bagel/Library/Application Support/virtualenv/seed-v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator [WARNING __main__:21]

Global Python 3 site (note: virtualenv is installed to my user directory using pip3)

bagel$ python3 -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python37.zip',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/lib-dynload',
    '/Users/bagel/Library/Python/3.7/lib/python/site-packages',
    '/usr/local/lib/python3.7/site-packages',
    '/usr/local/Cellar/python/3.7.6_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages',
]
USER_BASE: '/Users/bagel/Library/Python/3.7' (exists)
USER_SITE: '/Users/bagel/Library/Python/3.7/lib/python/site-packages' (exists)
ENABLE_USER_SITE: True

Global Python 2 site:

bagel$ python -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Library/Python/2.7/site-packages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
]
USER_BASE: '/Users/bagel/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/bagel/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

venv site:

(venv) bagel$ python -m site
sys.path = [
    '/Users/bagel/Documents/projects/test',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python27.zip',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-darwin',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/plat-mac/lib-scriptpackages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-tk',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-old',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload',
    '/Users/bagel/Documents/projects/test/venv/lib/python2.7/site-packages',
    '/Library/Python/2.7/site-packages',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python',
    '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC',
]
USER_BASE: '/Users/bagel/Library/Python/2.7' (doesn't exist)
USER_SITE: '/Users/bagel/Library/Python/2.7/lib/python/site-packages' (doesn't exist)
ENABLE_USER_SITE: False
gaborbernat commented 4 years ago

Oh another differently customized python under System path this time 🤷‍♂️macOs is wonderful. If you install brew version that will work.

gaborbernat commented 4 years ago

Note for some reason we're picking up /usr/bin/python2.7, not the global python you're showcasing.

rpkilby commented 4 years ago

For context I've installed Python 3.7 via brew, but Python 2.7 is what shipped with Mojave.

Note for some reason we're picking up /usr/bin/python2.7, not the global python you're showcasing.

It's just a symlink to the framework Python.

bagel$ ls /usr/bin/python2.7
lrwxr-xr-x 1 root wheel 75 Nov 16  2018 /usr/bin/python2.7 -> ../../System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7
gaborbernat commented 4 years ago

I'm fairly certain the issue here is that site.py of the shipped python adds the global python to the sys path unconditionally (ignoring blatantly the enable global site packages flag within site.py 🤦‍♂️🤷‍♂️ macOS) so you're picking up things as that. We'll need to escape this in the custom shim site.py we ship 👍

rpkilby commented 4 years ago

Idk if it helps, but here are the contents of the site module.

/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site.py ```python """Append module search paths for third-party packages to sys.path. **************************************************************** * This module is automatically imported during initialization. * **************************************************************** In earlier versions of Python (up to 1.5a3), scripts or modules that needed to use site-specific modules would place ``import site'' somewhere near the top of their code. Because of the automatic import, this is no longer necessary (but code that does it still works). This will append site-specific paths to the module search path. On Unix (including Mac OSX), it starts with sys.prefix and sys.exec_prefix (if different) and appends lib/python/site-packages as well as lib/site-python. On other platforms (such as Windows), it tries each of the prefixes directly, as well as with lib/site-packages appended. The resulting directories, if they exist, are appended to sys.path, and also inspected for path configuration files. A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line) to be added to sys.path. Non-existing directories (or non-directories) are never added to sys.path; no directory is added to sys.path more than once. Blank lines and lines beginning with '#' are skipped. Lines starting with 'import' are executed. For example, suppose sys.prefix and sys.exec_prefix are set to /usr/local and there is a directory /usr/local/lib/python2.5/site-packages with three subdirectories, foo, bar and spam, and two path configuration files, foo.pth and bar.pth. Assume foo.pth contains the following: # foo package configuration foo bar bletch and bar.pth contains: # bar package configuration bar Then the following directories are added to sys.path, in this order: /usr/local/lib/python2.5/site-packages/bar /usr/local/lib/python2.5/site-packages/foo Note that bletch is omitted because it doesn't exist; bar precedes foo because bar.pth comes alphabetically before foo.pth; and spam is omitted because it is not mentioned in either path configuration file. After these path manipulations, an attempt is made to import a module named sitecustomize, which can perform arbitrary additional site-specific customizations. If this import fails with an ImportError exception, it is silently ignored. """ import sys import os import __builtin__ import traceback # Prefixes for site-packages; add additional prefixes like /usr/local here PREFIXES = [sys.prefix, sys.exec_prefix] # Enable per user site-packages directory # set it to False to disable the feature or True to force the feature ENABLE_USER_SITE = None # for distutils.commands.install # These values are initialized by the getuserbase() and getusersitepackages() # functions, through the main() function when Python starts. USER_SITE = None USER_BASE = None def makepath(*paths): dir = os.path.join(*paths) try: dir = os.path.abspath(dir) except OSError: pass return dir, os.path.normcase(dir) def abs__file__(): """Set all module' __file__ attribute to an absolute path""" for m in sys.modules.values(): if hasattr(m, '__loader__'): continue # don't mess with a PEP 302-supplied __file__ try: m.__file__ = os.path.abspath(m.__file__) except (AttributeError, OSError): pass def removeduppaths(): """ Remove duplicate entries from sys.path along with making them absolute""" # This ensures that the initial path provided by the interpreter contains # only absolute pathnames, even if we're running from the build directory. L = [] known_paths = set() for dir in sys.path: # Filter out duplicate paths (on case-insensitive file systems also # if they only differ in case); turn relative paths into absolute # paths. dir, dircase = makepath(dir) if not dircase in known_paths: L.append(dir) known_paths.add(dircase) sys.path[:] = L return known_paths def _init_pathinfo(): """Return a set containing all existing directory entries from sys.path""" d = set() for dir in sys.path: try: if os.path.isdir(dir): dir, dircase = makepath(dir) d.add(dircase) except TypeError: continue return d def addpackage(sitedir, name, known_paths): """Process a .pth file within the site-packages directory: For each line in the file, either combine it with sitedir to a path and add that to known_paths, or execute it if it starts with 'import '. """ if known_paths is None: _init_pathinfo() reset = 1 else: reset = 0 fullname = os.path.join(sitedir, name) try: f = open(fullname, "rU") except IOError: return with f: for n, line in enumerate(f): if line.startswith("#"): continue try: if line.startswith(("import ", "import\t")): exec line continue line = line.rstrip() dir, dircase = makepath(sitedir, line) if not dircase in known_paths and os.path.exists(dir): sys.path.append(dir) known_paths.add(dircase) except Exception as err: print >>sys.stderr, "Error processing line {:d} of {}:\n".format( n+1, fullname) for record in traceback.format_exception(*sys.exc_info()): for line in record.splitlines(): print >>sys.stderr, ' '+line print >>sys.stderr, "\nRemainder of file ignored" break if reset: known_paths = None return known_paths def addsitedir(sitedir, known_paths=None): """Add 'sitedir' argument to sys.path if missing and handle .pth files in 'sitedir'""" if known_paths is None: known_paths = _init_pathinfo() reset = 1 else: reset = 0 sitedir, sitedircase = makepath(sitedir) if not sitedircase in known_paths: sys.path.append(sitedir) # Add path component try: names = os.listdir(sitedir) except os.error: return dotpth = os.extsep + "pth" names = [name for name in names if name.endswith(dotpth)] for name in sorted(names): addpackage(sitedir, name, known_paths) if reset: known_paths = None return known_paths def check_enableusersite(): """Check if user site directory is safe for inclusion The function tests for the command line flag (including environment var), process uid/gid equal to effective uid/gid. None: Disabled for security reasons False: Disabled by user (command line option) True: Safe and enabled """ if sys.flags.no_user_site: return False if hasattr(os, "getuid") and hasattr(os, "geteuid"): # check process uid == effective uid if os.geteuid() != os.getuid(): return None if hasattr(os, "getgid") and hasattr(os, "getegid"): # check process gid == effective gid if os.getegid() != os.getgid(): return None return True def getuserbase(): """Returns the `user base` directory path. The `user base` directory can be used to store data. If the global variable ``USER_BASE`` is not initialized yet, this function will also set it. """ global USER_BASE if USER_BASE is not None: return USER_BASE from sysconfig import get_config_var USER_BASE = get_config_var('userbase') return USER_BASE def getusersitepackages(): """Returns the user-specific site-packages directory path. If the global variable ``USER_SITE`` is not initialized yet, this function will also set it. """ global USER_SITE user_base = getuserbase() # this will also set USER_BASE if USER_SITE is not None: return USER_SITE from sysconfig import get_path import os if sys.platform == 'darwin': from sysconfig import get_config_var if get_config_var('PYTHONFRAMEWORK'): USER_SITE = get_path('purelib', 'osx_framework_user') return USER_SITE USER_SITE = get_path('purelib', '%s_user' % os.name) return USER_SITE def addusersitepackages(known_paths): """Add a per user site-package to sys.path Each user has its own python directory with site-packages in the home directory. """ # get the per user site-package path # this call will also make sure USER_BASE and USER_SITE are set user_site = getusersitepackages() if ENABLE_USER_SITE and os.path.isdir(user_site): addsitedir(user_site, known_paths) return known_paths def getsitepackages(): """Returns a list containing all global site-packages directories (and possibly site-python). For each directory present in the global ``PREFIXES``, this function will find its `site-packages` subdirectory depending on the system environment, and will return a list of full paths. """ sitepackages = [] seen = set() for prefix in PREFIXES: if not prefix or prefix in seen: continue seen.add(prefix) if sys.platform in ('os2emx', 'riscos'): sitepackages.append(os.path.join(prefix, "Lib", "site-packages")) elif os.sep == '/': sitepackages.append(os.path.join(prefix, "lib", "python" + sys.version[:3], "site-packages")) sitepackages.append(os.path.join(prefix, "lib", "site-python")) else: sitepackages.append(prefix) sitepackages.append(os.path.join(prefix, "lib", "site-packages")) if sys.platform == "darwin": # for framework builds *only* we add the standard Apple # locations. from sysconfig import get_config_var framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages")) return sitepackages def addsitepackages(known_paths): """Add site-packages (and possibly site-python) to sys.path""" for sitedir in getsitepackages(): if os.path.isdir(sitedir): addsitedir(sitedir, known_paths) return known_paths def setBEGINLIBPATH(): """The OS/2 EMX port has optional extension modules that do double duty as DLLs (and must use the .DLL file extension) for other extensions. The library search path needs to be amended so these will be found during module import. Use BEGINLIBPATH so that these are at the start of the library search path. """ dllpath = os.path.join(sys.prefix, "Lib", "lib-dynload") libpath = os.environ['BEGINLIBPATH'].split(';') if libpath[-1]: libpath.append(dllpath) else: libpath[-1] = dllpath os.environ['BEGINLIBPATH'] = ';'.join(libpath) def setquit(): """Define new builtins 'quit' and 'exit'. These are objects which make the interpreter exit when called. The repr of each object contains a hint at how it works. """ if os.sep == ':': eof = 'Cmd-Q' elif os.sep == '\\': eof = 'Ctrl-Z plus Return' else: eof = 'Ctrl-D (i.e. EOF)' class Quitter(object): def __init__(self, name): self.name = name def __repr__(self): return 'Use %s() or %s to exit' % (self.name, eof) def __call__(self, code=None): # Shells like IDLE catch the SystemExit, but listen when their # stdin wrapper is closed. try: sys.stdin.close() except: pass raise SystemExit(code) __builtin__.quit = Quitter('quit') __builtin__.exit = Quitter('exit') class _Printer(object): """interactive prompt objects for printing the license text, a list of contributors and the copyright notice.""" MAXLINES = 23 def __init__(self, name, data, files=(), dirs=()): self.__name = name self.__data = data self.__files = files self.__dirs = dirs self.__lines = None def __setup(self): if self.__lines: return data = None for dir in self.__dirs: for filename in self.__files: filename = os.path.join(dir, filename) try: fp = file(filename, "rU") data = fp.read() fp.close() break except IOError: pass if data: break if not data: data = self.__data self.__lines = data.split('\n') self.__linecnt = len(self.__lines) def __repr__(self): self.__setup() if len(self.__lines) <= self.MAXLINES: return "\n".join(self.__lines) else: return "Type %s() to see the full %s text" % ((self.__name,)*2) def __call__(self): self.__setup() prompt = 'Hit Return for more, or q (and Return) to quit: ' lineno = 0 while 1: try: for i in range(lineno, lineno + self.MAXLINES): print self.__lines[i] except IndexError: break else: lineno += self.MAXLINES key = None while key is None: key = raw_input(prompt) if key not in ('', 'q'): key = None if key == 'q': break def setcopyright(): """Set 'copyright' and 'credits' in __builtin__""" __builtin__.copyright = _Printer("copyright", sys.copyright) if sys.platform[:4] == 'java': __builtin__.credits = _Printer( "credits", "Jython is maintained by the Jython developers (www.jython.org).") else: __builtin__.credits = _Printer("credits", """\ Thanks to CWI, CNRI, BeOpen.com, Zope Corporation and a cast of thousands for supporting Python development. See www.python.org for more information.""") here = os.path.dirname(os.__file__) __builtin__.license = _Printer( "license", "See https://www.python.org/psf/license/", ["LICENSE.txt", "LICENSE"], [os.path.join(here, os.pardir), here, os.curdir]) class _Helper(object): """Define the builtin 'help'. This is a wrapper around pydoc.help (with a twist). """ def __repr__(self): return "Type help() for interactive help, " \ "or help(object) for help about object." def __call__(self, *args, **kwds): import pydoc return pydoc.help(*args, **kwds) def sethelper(): __builtin__.help = _Helper() def aliasmbcs(): """On Windows, some default encodings are not provided by Python, while they are always available as "mbcs" in each locale. Make them usable by aliasing to "mbcs" in such a case.""" if sys.platform == 'win32': import locale, codecs enc = locale.getdefaultlocale()[1] if enc.startswith('cp'): # "cp***" ? try: codecs.lookup(enc) except LookupError: import encodings encodings._cache[enc] = encodings._unknown encodings.aliases.aliases[enc] = 'mbcs' def setencoding(): """Set the string encoding used by the Unicode implementation. The default is 'ascii', but if you're willing to experiment, you can change this.""" encoding = "ascii" # Default value set by _PyUnicode_Init() if 0: # Enable to support locale aware default string encodings. import locale loc = locale.getdefaultlocale() if loc[1]: encoding = loc[1] if 0: # Enable to switch off string to Unicode coercion and implicit # Unicode to string conversion. encoding = "undefined" if encoding != "ascii": # On Non-Unicode builds this will raise an AttributeError... sys.setdefaultencoding(encoding) # Needs Python Unicode build ! def execsitecustomize(): """Run custom site specific code, if available.""" try: import sitecustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print >>sys.stderr, \ "'import sitecustomize' failed; use -v for traceback" def execusercustomize(): """Run custom user specific code, if available.""" try: import usercustomize except ImportError: pass except Exception: if sys.flags.verbose: sys.excepthook(*sys.exc_info()) else: print>>sys.stderr, \ "'import usercustomize' failed; use -v for traceback" def main(): global ENABLE_USER_SITE abs__file__() known_paths = removeduppaths() if ENABLE_USER_SITE is None: ENABLE_USER_SITE = check_enableusersite() known_paths = addusersitepackages(known_paths) known_paths = addsitepackages(known_paths) if sys.platform == 'os2emx': setBEGINLIBPATH() setquit() setcopyright() sethelper() aliasmbcs() setencoding() execsitecustomize() if ENABLE_USER_SITE: execusercustomize() # Remove sys.setdefaultencoding() so that users cannot change the # encoding after initialization. The test for presence is needed when # this module is run as a script, because this code is executed twice. if hasattr(sys, "setdefaultencoding"): del sys.setdefaultencoding main() def _script(): help = """\ %s [--user-base] [--user-site] Without arguments print some useful information With arguments print the value of USER_BASE and/or USER_SITE separated by '%s'. Exit codes with --user-base or --user-site: 0 - user site directory is enabled 1 - user site directory is disabled by user 2 - uses site directory is disabled by super user or for security reasons >2 - unknown error """ args = sys.argv[1:] if not args: print "sys.path = [" for dir in sys.path: print " %r," % (dir,) print "]" print "USER_BASE: %r (%s)" % (USER_BASE, "exists" if os.path.isdir(USER_BASE) else "doesn't exist") print "USER_SITE: %r (%s)" % (USER_SITE, "exists" if os.path.isdir(USER_SITE) else "doesn't exist") print "ENABLE_USER_SITE: %r" % ENABLE_USER_SITE sys.exit(0) buffer = [] if '--user-base' in args: buffer.append(USER_BASE) if '--user-site' in args: buffer.append(USER_SITE) if buffer: print os.pathsep.join(buffer) if ENABLE_USER_SITE: sys.exit(0) elif ENABLE_USER_SITE is False: sys.exit(1) elif ENABLE_USER_SITE is None: sys.exit(2) else: sys.exit(3) else: import textwrap print textwrap.dedent(help % (sys.argv[0], os.pathsep)) sys.exit(10) if __name__ == '__main__': _script() ```
gaborbernat commented 4 years ago

Yeah see the hardcoded part at

framework = get_config_var("PYTHONFRAMEWORK") if framework: sitepackages.append( os.path.join("/Library", framework, sys.version[:3], "site-packages"

YannickJadoul commented 4 years ago

@gaborbernat I've managed to reproduce my remaining issue in a simpler way:

mkdir tmp_dir
cd tmp_dir

curl -L https://bitbucket.org/pypy/pypy/downloads/pypy2.7-v7.3.0-osx64.tar.bz2 | tar xj
curl -L https://bitbucket.org/pypy/pypy/downloads/pypy3.6-v7.3.0-osx64.tar.bz2 | tar xj
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py

python3 -m virtualenv --version
python3 -m virtualenv env
source env/bin/activate

pip install pytest

cat << EOF > test.py
import os
import subprocess

def test2():
    subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', 'get-pip.py', '-v'])
    subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', '-m', 'site'])
    print(os.listdir('pypy2.7-v7.3.0-osx64/bin'))
    assert False

def test3():
    subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', 'get-pip.py', '-v'])
    subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', '-m', 'site'])
    print(os.listdir('pypy3.6-v7.3.0-osx64/bin'))
    assert False

EOF

pytest test.py
Output

``` ai31:~ yannick$ ./minimal_example.sh % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 20.5M 100 20.5M 0 0 1263k 0 0:00:16 0:00:16 --:--:-- 1707k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 20.2M 100 20.2M 0 0 1406k 0 0:00:14 0:00:14 --:--:-- 1607k % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 1764k 100 1764k 0 0 6561k 0 --:--:-- --:--:-- --:--:-- 6561k virtualenv 20.0.5 from /usr/local/lib/python3.7/site-packages/virtualenv/__init__.py created virtual environment CPython3.7.5.final.0-64 in 199ms creator CPython3Posix(dest=/Users/yannick/tmp_dir/env, clear=False, global=False) seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/yannick/Library/Application Support/virtualenv/seed-v1) activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator Collecting pytest Using cached pytest-5.3.5-py3-none-any.whl (235 kB) Collecting packaging Using cached packaging-20.1-py2.py3-none-any.whl (36 kB) Collecting importlib-metadata>=0.12; python_version < "3.8" Using cached importlib_metadata-1.5.0-py2.py3-none-any.whl (30 kB) Collecting more-itertools>=4.0.0 Using cached more_itertools-8.2.0-py3-none-any.whl (43 kB) Collecting py>=1.5.0 Using cached py-1.8.1-py2.py3-none-any.whl (83 kB) Collecting pluggy<1.0,>=0.12 Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB) Collecting attrs>=17.4.0 Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB) Collecting wcwidth Using cached wcwidth-0.1.8-py2.py3-none-any.whl (17 kB) Collecting pyparsing>=2.0.2 Using cached pyparsing-2.4.6-py2.py3-none-any.whl (67 kB) Collecting six Using cached six-1.14.0-py2.py3-none-any.whl (10 kB) Collecting zipp>=0.5 Using cached zipp-3.0.0-py3-none-any.whl (4.8 kB) Installing collected packages: pyparsing, six, packaging, zipp, importlib-metadata, more-itertools, py, pluggy, attrs, wcwidth, pytest Successfully installed attrs-19.3.0 importlib-metadata-1.5.0 more-itertools-8.2.0 packaging-20.1 pluggy-0.13.1 py-1.8.1 pyparsing-2.4.6 pytest-5.3.5 six-1.14.0 wcwidth-0.1.8 zipp-3.0.0 ======================================== test session starts ======================================== platform darwin -- Python 3.7.5, pytest-5.3.5, py-1.8.1, pluggy-0.13.1 rootdir: /Users/yannick/tmp_dir collected 2 items test.py FF [100%] ============================================= FAILURES ============================================== _______________________________________________ test2 _______________________________________________ def test2(): subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', 'get-pip.py', '-v']) subprocess.check_call(['pypy2.7-v7.3.0-osx64/bin/pypy', '-m', 'site']) print(os.listdir('pypy2.7-v7.3.0-osx64/bin')) > assert False E assert False test.py:8: AssertionError --------------------------------------- Captured stdout call ---------------------------------------- Non-user install because site-packages writeable Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-ephem-wheel-cache-Dzm69W Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d Initialized build tracking at /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d Created build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d Entered build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-install-yhafaR 1 location(s) to search for versions of pip: * https://pypi.org/simple/pip/ Fetching project page and analyzing links: https://pypi.org/simple/pip/ Getting page https://pypi.org/simple/pip/ Found index url https://pypi.org/simple Looking up "https://pypi.org/simple/pip/" in the cache Request header has "max_age" as 0, cache bypassed Starting new HTTPS connection (1): pypi.org:443 https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0 Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2 ... Installing collected packages: pip, setuptools, wheel Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-xnir60 changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip to 755 changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip2 to 755 changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/pip2.7 to 755 Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-qGsleO changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/easy_install to 755 changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/easy_install-2.7 to 755 Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-erEv1A changing mode of /Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/bin/wheel to 755 Successfully installed pip-20.0.2 setuptools-44.0.0 wheel-0.34.2 Cleaning up... Removed build tracker: '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-4kc17d' sys.path = [ '/Users/yannick/tmp_dir', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib_pypy/__extensions__', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib_pypy', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/lib-tk', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-darwin', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-mac', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/lib-python/2.7/plat-mac/lib-scriptpackages', '/Users/yannick/tmp_dir/pypy2.7-v7.3.0-osx64/site-packages', ] USER_BASE: '/Users/yannick/.local' (doesn't exist) USER_SITE: '/Users/yannick/.local/lib/pypy2.7/site-packages' (doesn't exist) ENABLE_USER_SITE: True ['libpypy-c.dylib', 'pip2', 'easy_install', 'wheel', 'pip', 'pypy', 'easy_install-2.7', 'pip2.7'] --------------------------------------- Captured stderr call ---------------------------------------- DEPRECATION: A future version of pip will drop support for Python 2.7. More details about Python 2 support in pip, can be found at https://pip.pypa.io/en/latest/development/release-process/#python-2-support _______________________________________________ test3 _______________________________________________ def test3(): subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', 'get-pip.py', '-v']) subprocess.check_call(['pypy3.6-v7.3.0-osx64/bin/pypy3', '-m', 'site']) print(os.listdir('pypy3.6-v7.3.0-osx64/bin')) > assert False E assert False test.py:14: AssertionError --------------------------------------- Captured stdout call ---------------------------------------- Non-user install because user site-packages disabled Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-ephem-wheel-cache-p_j2cq6p Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b Initialized build tracking at /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b Created build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b Entered build tracker: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-install-08qoikav 1 location(s) to search for versions of pip: * https://pypi.org/simple/pip/ Fetching project page and analyzing links: https://pypi.org/simple/pip/ Getting page https://pypi.org/simple/pip/ Found index url https://pypi.org/simple Looking up "https://pypi.org/simple/pip/" in the cache Request header has "max_age" as 0, cache bypassed Starting new HTTPS connection (1): pypi.org:443 https://pypi.org:443 "GET /simple/pip/ HTTP/1.1" 304 0 Found link https://files.pythonhosted.org/packages/3d/9d/1e313763bdfb6a48977b65829c6ce2a43eaae29ea2f907c8bbef024a7219/pip-0.2.tar.gz#sha256=88bb8d029e1bf4acd0e04d300104b7440086f94cc1ce1c5c3c31e3293aee1f81 (from https://pypi.org/simple/pip/), version: 0.2 ... Installing collected packages: pip, setuptools, wheel Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-_wruh4x9 changing mode of /Users/yannick/tmp_dir/env/bin/pip to 755 changing mode of /Users/yannick/tmp_dir/env/bin/pip3 to 755 changing mode of /Users/yannick/tmp_dir/env/bin/pip3.6 to 755 Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-15knx9f1 changing mode of /Users/yannick/tmp_dir/env/bin/easy_install to 755 changing mode of /Users/yannick/tmp_dir/env/bin/easy_install-3.6 to 755 Created temporary directory: /private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-unpacked-wheel-n8_gcseu changing mode of /Users/yannick/tmp_dir/env/bin/wheel to 755 Successfully installed pip-20.0.2 setuptools-45.2.0 wheel-0.34.2 Cleaning up... Removed build tracker: '/private/var/folders/2k/7lc28xfj6vbfpyjpphcvyh7w0000gn/T/pip-req-tracker-na7rsa2b' sys.path = [ '/Users/yannick/tmp_dir', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac', '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages', '/Users/yannick/tmp_dir/env/site-packages', ] USER_BASE: '/Users/yannick/.local' (doesn't exist) USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist) ENABLE_USER_SITE: False ['libpypy3-c.dylib', 'pypy3'] ======================================== 2 failed in 23.66s ========================================= ```

I think the main thing to notice in that whole output, is that when running PyPy 3.6, v7.3.0 from within pytest from within a virtualenv (oh yes, and somehow only on macOS; and oh yes, on virtualenv<20, it works), -m site gives this output:

sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/env/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: False

while if I afterwards run the same command outside of pytest, I get:

ai31:~ yannick$ cd tmp_dir/
ai31:tmp_dir yannick$ source env/bin/activate
(env) ai31:tmp_dir yannick$ pypy3.6-v7.3.0-osx64/bin/pypy3 -m site
sys.path = [
    '/Users/yannick/tmp_dir',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy/__extensions__',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib_pypy',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/lib-tk',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-darwin',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/lib-python/3/plat-mac/lib-scriptpackages',
    '/Users/yannick/tmp_dir/pypy3.6-v7.3.0-osx64/site-packages',
]
USER_BASE: '/Users/yannick/.local' (doesn't exist)
USER_SITE: '/Users/yannick/.local/lib/pypy3.6/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

Somehow, the same doesn't happen with PyPy 2.7.

EDIT: I'm really sorry for finding such a nasty bug, that needs to be reproduced in such a horrible way :-/

gaborbernat commented 4 years ago

@rpkilby your issue should be addressed with https://github.com/pypa/virtualenv/pull/1653, can you validate? Thanks!

gaborbernat commented 4 years ago

@YannickJadoul you're issue is related with pypy, please open a separate issue for that; with a clearer description of what's expected and what happens 👍 this issue is mainly about CPython Framework builds on macOs.

YannickJadoul commented 4 years ago

@YannickJadoul you're issue is related with pypy, please open a separate issue for that; with a clearer description of what's expected and what happens +1 this issue is mainly about CPython Framework builds on macOs.

Yeah, sorry, I came across these two issues at the same time, so I kind of assumed they were linked. I've opened #1654 now, hopefully that's more clear :-)

rpkilby commented 4 years ago

@gaborbernat #1653 seems to work :D

gaborbernat commented 4 years ago

Hello, a fix for this issue has been released via virtualenv 20.0.6; see https://pypi.org/project/virtualenv/20.0.6/ (https://virtualenv.pypa.io/en/latest/changelog.html#v20-0-6-2020-02-26) . Please give a try and report back if your issue has not been addressed; if not, please comment here, and we'll reopen the ticket. We want to apologize for the inconvenience this has caused you and say thanks for having patience while we resolve the unexpected bugs with this new major release. thanks