search

uqfoundation / pathos

parallel graph management and execution in heterogeneous computing
http://pathos.rtfd.io
Other
1.38k stars 89 forks source link

installs wrong multiprocess wheel for pypy #222

Closed mmckerns closed 2 years ago

mmckerns commented 2 years ago

Travis CI installs the wrong multiprocess wheel when for pypy3.6 and pypy3.7. A pathos pypy3.6 install identifies multiprocess-0.70.12.2-py36-none-any.whl as a dependency

Installed /home/travis/virtualenv/pypy3.6-7.3.3/site-packages/pathos-0.2.9.dev0-py3.6.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/df/d0/c3011a5bb77f307e68682a5046cce1a2c6591267bf24b5bf3fc4130bb39d/multiprocess-0.70.12.2-py36-none-any.whl#sha256=b3f866f7d9c7acc1a9cb1b6063a29f5cb140ff545b35b71fd4bfdac6f19d75fa
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2-py36-none-any.whl
Installing multiprocess-0.70.12.2-py36-none-any.whl to /home/travis/virtualenv/pypy3.6-7.3.3/site-packages
Adding multiprocess 0.70.12.2 to easy-install.pth file

and a pathos pypy3.7 install also identifies multiprocess-0.70.12.2-py36-none-any.whl

Installed /home/travis/virtualenv/pypy3.7-7.3.5/site-packages/pathos-0.2.9.dev0-py3.7.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/df/d0/c3011a5bb77f307e68682a5046cce1a2c6591267bf24b5bf3fc4130bb39d/multiprocess-0.70.12.2-py36-none-any.whl#sha256=b3f866f7d9c7acc1a9cb1b6063a29f5cb140ff545b35b71fd4bfdac6f19d75fa
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2-py36-none-any.whl
Installing multiprocess-0.70.12.2-py36-none-any.whl to /home/travis/virtualenv/pypy3.7-7.3.5/site-packages
Adding multiprocess 0.70.12.2 to easy-install.pth file

Note that both do not install the pypy-specific versions (i.e. multiprocess-0.70.12.2-pp36-none-any.whl), and the latter also installs the wrong version of python (i.e. 3.6 instead of 3.7).

mmckerns commented 2 years ago

For reference, see: https://app.travis-ci.com/github/uqfoundation/pathos/builds/239671751

mmckerns commented 2 years ago

a pathos pypy2.7 also installs the wrong multiprocess dependency (zip not whl):

Installed /home/travis/virtualenv/pypy2.7-7.3.1/site-packages/pathos-0.2.9.dev0-py2.7.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/99/1a/472900644359cdd208d1fbe71706bdeecbc6e8db2e39c35ebe89459e9172/multiprocess-0.70.12.2.zip#sha256=206bb9b97b73f87fec1ed15a19f8762950256aa84225450abc7150d02855a083
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2.zip
Writing /tmp/easy_install-ZZ7SxN/multiprocess-0.70.12.2/setup.cfg
Running multiprocess-0.70.12.2/setup.py -q bdist_egg --dist-dir /tmp/easy_install-ZZ7SxN/multiprocess-0.70.12.2/egg-dist-tmp-aEPlAM
no previously-included directories found matching '.git'
no previously-included directories found matching '.coverage'
creating /home/travis/virtualenv/pypy2.7-7.3.1/site-packages/multiprocess-0.70.12.2-py2.7.egg
Extracting multiprocess-0.70.12.2-py2.7.egg to /home/travis/virtualenv/pypy2.7-7.3.1/site-packages
Adding multiprocess 0.70.12.2 to easy-install.pth file
mmckerns commented 2 years ago

Seems similar issue exists for a local install as well...

Installed /Users/mmckerns/lib/pypy-2.7/site-packages/pathos-0.2.9.dev0-py2.7.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/99/1a/472900644359cdd208d1fbe71706bdeecbc6e8db2e39c35ebe89459e9172/multiprocess-0.70.12.2.zip#sha256=206bb9b97b73f87fec1ed15a19f8762950256aa84225450abc7150d02855a083
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2.zip
...
mmckerns commented 2 years ago

Same result for local install with pypy3.6

Installed /Users/mmckerns/lib/pypy-3.6/site-packages/pathos-0.2.9.dev0-py3.6.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/df/d0/c3011a5bb77f307e68682a5046cce1a2c6591267bf24b5bf3fc4130bb39d/multiprocess-0.70.12.2-py36-none-any.whl#sha256=b3f866f7d9c7acc1a9cb1b6063a29f5cb140ff545b35b71fd4bfdac6f19d75fa
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2-py36-none-any.whl
...

and pypy3.7

Installed /Users/mmckerns/lib/pypy-3.7/site-packages/pathos-0.2.9.dev0-py3.7.egg
Processing dependencies for pathos==0.2.9.dev0
Searching for multiprocess>=0.70.12.1
Reading https://pypi.org/simple/multiprocess/
Downloading https://files.pythonhosted.org/packages/df/d0/c3011a5bb77f307e68682a5046cce1a2c6591267bf24b5bf3fc4130bb39d/multiprocess-0.70.12.2-py36-none-any.whl#sha256=b3f866f7d9c7acc1a9cb1b6063a29f5cb140ff545b35b71fd4bfdac6f19d75fa
Best match: multiprocess 0.70.12.2
Processing multiprocess-0.70.12.2-py36-none-any.whl
...
mmckerns commented 2 years ago

Possibly due to badly named multiprocess wheel... however, pypy3.7 installing a py3.6 wheel is very odd.

mmckerns commented 2 years ago

It appears as pip moved from 19.x to 20.x, the supported naming conventions for pypy changed: https://github.com/pypa/packaging/issues/311 https://github.com/pypa/pip/issues/7629

Thus pp37-none-any.whl (used by multiprocess) is no longer supported:

$ pypy3.7
>>>> from packaging import tags
>>>> for t in tags.generic_tags(): print(t)
pp37-pypy37_pp73-macosx_***
[SNIPPED]
pp37-none-macosx_***
[SNIPPED]
>>>> for t in tags.compatible_tags(): print(t)
[SNIPPED]
py37-none-any
py3-none-any
py36-none-any
py35-none-any
py34-none-any
py33-none-any
py32-none-any
py31-none-any
py30-none-any

Same for pp36-none-any.whl and pp27-none-any.whl.

mmckerns commented 2 years ago

Workaround added for travis (force a source install): 835a3fd7e4583e854f75503670e7bf3610e842d3. The issue still remains for a non-travis install.

mmckerns commented 2 years ago

should be fixed by the release of multiprocess-0.70.13