search

NNPDF / pineappl

PineAPPL is not an extension of APPLgrid
https://nnpdf.github.io/pineappl/
GNU General Public License v3.0
12 stars 3 forks source link

Fix python version problems #214

Closed cschwan closed 1 year ago

cschwan commented 1 year ago

Here we'll fix the problems discovered in this comment: https://github.com/NNPDF/pineappl/issues/211#issuecomment-1453282553.

cschwan commented 1 year ago

It seems to work, except that on Windows no PyPy (3.7, 3.8, 3.9) wheels are created.

alecandido commented 1 year ago

It seems to work, except that on Windows no PyPy (3.7, 3.8, 3.9) wheels are created.

That is not a problem :)

alecandido commented 1 year ago

I still don't see MacOS wheels for CPython 3.7, 3.8, and 3.9 in https://github.com/NNPDF/pineappl/suites/11369716006/artifacts/584873397 (i.e. the wheels archive from https://github.com/NNPDF/pineappl/actions/runs/4341762408)

cschwan commented 1 year ago

You're right: instead the corresponding PyPy wheels are created (and also found) twice. See this line from the action:

Found PyPy 3.7 at /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/python3.7, PyPy 3.8 at /Users/runner/hostedtoolcache/PyPy/3.8.16/x64/bin/python3.8, PyPy 3.9 at /Users/runner/hostedtoolcache/PyPy/3.9.16/x64/bin/python3.9, CPython 3.10 at /Users/runner/hostedtoolcache/Python/3.10.10/x64/bin/python3.10, CPython 3.11 at /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11, PyPy 3.7 at /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/pypy3.7, PyPy 3.8 at /Users/runner/hostedtoolcache/PyPy/3.8.16/x64/bin/pypy3.8, PyPy 3.9 at /Users/runner/hostedtoolcache/PyPy/3.9.16/x64/bin/pypy3.9

Note how it finds PyPy (?) 3.7

Both claim to have

Built wheel for PyPy 3.7 to dist/pineappl-0.6.0a2-pp37-pypy37_pp73-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl
cschwan commented 1 year ago

It seems to work, except that on Windows no PyPy (3.7, 3.8, 3.9) wheels are created.

That is not a problem :)

I agree, but what worries me is that I don't understand what's going on here. ~I specifically requested these versions, why are they not present?~

They are installed, but it seems maturin doesn't find them:

Found CPython 3.11 at C:\hostedtoolcache\windows\Python\3.11.2\x64\python.exe, CPython 3.10 at C:\hostedtoolcache\windows\Python\3.10.10\x64\python.exe, CPython 3.9 at C:\hostedtoolcache\windows\Python\3.9.13\x64\python.exe, CPython 3.8 at C:\hostedtoolcache\windows\Python\3.8.10\x64\python.exe, CPython 3.7 at C:\hostedtoolcache\windows\Python\3.7.9\x64\python.exe
alecandido commented 1 year ago

They are installed, but it seems maturin doesn't find them:

Then there are two main options:

  1. try specifying everything explicitly, and giving up with --find-interpreter (at least in this case)
  2. file an issue on maturin repo (messense is usually replying fast and very helpful)

Note how it finds PyPy (?) 3.7

  • once as /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/python3.7 and
  • then as /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/pypy3.7.

This I believe to be normal, and most likely the first is a link to the second, in such a way that PyPy can work both side-by-side and as a drop-in replacement of CPython.

cschwan commented 1 year ago

Note how it finds PyPy (?) 3.7

  • once as /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/python3.7 and
  • then as /Users/runner/hostedtoolcache/PyPy/3.7.13/x64/bin/pypy3.7.

This I believe to be normal, and most likely the first is a link to the second, in such a way that PyPy can work both side-by-side and as a drop-in replacement of CPython.

Is this OK? Because the wheel is twice created for PyPy, can CPython work with it?

alecandido commented 1 year ago

Is this OK? Because the wheel is twice created for PyPy, can CPython work with it?

No, definitely not. The compiled wheel needs ABI compatibility with the interpreter. In order to guarantee this, wheels adopted a consistent naming scheme, that is the one you observed in the generated objects.

It might be that on MacOS, for whatever reason, PyPy is actually the python3.x x \in [7, 8, 9] executable found by maturin, so it is regenerating twice the wheel for PyPy and overwriting it, while those for CPython are never generated.

Since for PyPy Linux is definitely far more than enough, maybe just try to remove it from all the lists of python-version (for Linux it will be built anyhow, since PyPy will be inside the container).

alecandido commented 1 year ago

@cschwan I have the feeling that 80433c073efd0f825cf8582892c16c10895dddb0 is in the wrong branch, but I might be wrong...

EDIT: ah sorry, I just got the notification missed the series of "Revert" commits; we will squash anyhow, so please ignore my message

alecandido commented 1 year ago

@cschwan in https://github.com/NNPDF/pineappl/actions/runs/4344435880 we actually obtained all the wheels we wanted:

Moreover, the workflow is now running by default on push:, but without uploading the wheels (that should only happen on releases, but of course this I couldn't test). Everything is documented in update-wheels.sh.

Should we agree on the current result, or do we have still some desiderata left behind?

alecandido commented 1 year ago

Curious fact: Windows has the smallest wheels, while MacOS the biggest ones

The size of the wheel does not depend on the Python version or the interpreter (wheels for Linux-PyPy are the same size of Linux-CPython). This makes me believe that they are actually very similar, and only the platform makes an actual significant difference...

cschwan commented 1 year ago

Should we agree on the current result, or do we have still some desiderata left behind?

As far as it concerns me, that's perfect! Go ahead and merge it.

cschwan commented 1 year ago

Curious fact: Windows has the smallest wheels, while MacOS the biggest ones

* Win: 676 KB

* Linux: 764 KB

* MacOS: 1.3 MB

The size of the wheel does not depend on the Python version or the interpreter (wheels for Linux-PyPy are the same size of Linux-CPython). This makes me believe that they are actually very similar, and only the platform makes an actual significant difference...

These numbers make sense, because for MacOS the universal wheels are basically two wheels in one, one for x86_64 and the other for aarch64.