PyO3 / maturin-action

GitHub Action to install and run a custom maturin command with built-in support for cross compilation
MIT License
132 stars 37 forks source link

cffi install inside manylinux docker ignores interpreter argument #164

Closed jjfiv closed 1 year ago

jjfiv commented 1 year ago

Hi all,

Thank you for this action and the great tooling. In trying to modernize my project's setup (which uses cffi bindings) I bumped into an issue in the install of maturin on the manylinux docker image. While the code emits an install of cffi, it does so using the python3 interpreter, rather than the interpreter set by -i, which is intended to be passed to maturin.

The logs from the Install Maturin Step look like:

    % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed

    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0

  100 10.3M  100 10.3M    0     0  13.5M      0 --:--:-- --:--:-- --:--:-- 13.5M
  maturin 0.14.15
  Collecting cffi
    Downloading cffi-1.15.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 427.9/427.9 kB 7.7 MB/s eta 0:00:00
  Collecting pycparser
    Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 32.5 MB/s eta 0:00:00
  Installing collected packages: pycparser, cffi
  Successfully installed cffi-1.15.1 pycparser-2.21
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

Note the cp3.7: cffi-1.15.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

... which invariably led to the target interpreter (e.g., python3.9) unable to find cffi at a critical moment:

     Finished release [optimized] target(s) in 49.24s
💥 maturin failed
  Caused by: Failed to generate cffi declarations using /usr/local/bin/python3.9: exit status: 1
--- Stdout:

--- Stderr:
Traceback (most recent call last):
  File "<string>", line 2, in <module>
ModuleNotFoundError: No module named 'cffi'

The critical line seems to be: https://github.com/PyO3/maturin-action/blob/a0af1065e1feac94a02f140de4bb6adfe9873385/src/index.ts#L481

It looks like the zig and sccache support might be similarly affected.

I was able to workaround this issue with a before-script-linux:

      - name: Build Wheels
        uses: PyO3/maturin-action@v1
        with:
          target: ${{ matrix.target }}
          manylinux: auto
          before-script-linux: python${{ matrix.python-version }} -m pip install cffi
          args: -i python${{ matrix.python-version }} --release --out dist

I don't know much about github action internals, but it looks like this is fairly tricky to fix because you'd have to potentially parse the args array, which may not be something that you want to do in the action, since you're not doing so for any other reason.

Potentially another way to solve this might be to take an interpreter as a separate yaml/parameter to this action, but I don't know the larger implications of that.

messense commented 1 year ago

Thanks for the detailed report!

cffi doesn't require building for every supported python interpreter, so you can just remove the -i option.