search

piwheels / packages

Issue tracker for piwheels package issues
https://github.com/piwheels/packages/issues
20 stars 5 forks source link

Package issue: cryptography 41.0.3 leads to cffi 1.15.1 failure on "32-bit" Raspberry Pi OS [REASON: /etc/pip.conf missing on some Bookworm pre-releases] #390

Closed holta closed 9 months ago

holta commented 9 months ago

Package name

cryptography & cffi

Package version

41.0.3 & 1.15.1

PyPI URL

https://pypi.org/project/cryptography/ & https://pypi.org/project/cffi/

piwheels URL

https://www.piwheels.org/project/cryptography/ & https://www.piwheels.org/project/cffi/

Python version

I am the maintainer

More information

Just FYI cffi (C Foreign Function Interface) failure occurs even with the new Python 3.11 wheels (thank you!!) that were just recently posted here:

Error is pasted in below in the hopes that this helps us & all move the ball forward as Raspberry Pi OS "Bookworm" converges soon. :pray:

FYI the error occurs regardless whether running with a 32-bit kernel or 64-bit kernel — on a fresh "officially 32-bit" RasPiOS 12 Lite (Bookworm, Python 3.11) as apt updated from https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2023-05-03/2023-05-03-raspios-bullseye-armhf-lite.img.xz — i.e. regardless whether or not Raspberry Pi OS's newer override arm_64bit=0 is set in /boot/config.txt (raspberrypi/firmware#1795) to force the running of a true 32-bit kernel.

OS/Architecture/Prep Details ``` root@raspberrypi:~# cat /etc/issue Raspbian GNU/Linux 12 \n \l root@raspberrypi:~# cat /etc/rpi-issue Raspberry Pi reference 2023-05-03 Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 47eee1f0ddcf8811559d51eea1c1bb48335e3e88, stage2 root@raspberrypi:~# grep -r bookworm /etc/apt /etc/apt/sources.list.d/raspi.list:deb http://archive.raspberrypi.org/debian/ bookworm main /etc/apt/sources.list.d/raspi.list:#deb-src http://archive.raspberrypi.org/debian/ bookworm main /etc/apt/sources.list:deb http://raspbian.raspberrypi.org/raspbian/ bookworm main contrib non-free rpi /etc/apt/sources.list:#deb-src http://raspbian.raspberrypi.org/raspbian/ bookworm main contrib non-free rpi root@raspberrypi:~# uname -a Linux raspberrypi 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux root@raspberrypi:~# tail -1 /proc/cpuinfo Model : Raspberry Pi 4 Model B Rev 1.1 root@raspberrypi:~# apt install python3-venv Reading package lists... Done Building dependency tree... Done Reading state information... Done The following additional packages will be installed: python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3.11-venv The following NEW packages will be installed: python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3-venv python3.11-venv 0 upgraded, 6 newly installed, 0 to remove and 0 not upgraded. Need to get 3,043 kB of archives. After this operation, 4,197 kB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-lib2to3 all 3.11.2-3 [76.3 kB] Get:2 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-distutils all 3.11.2-3 [131 kB] Get:3 http://raspbian.raspberrypi.org/raspbian bookworm/main armhf python3-pip-whl all 23.0.1+dfsg-1 [1,717 kB] Get:4 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3-setuptools-whl all 66.1.1-1 [1,111 kB] Get:6 http://raspbian.raspberrypi.org/raspbian bookworm/main armhf python3-venv armhf 3.11.2-1 [1,188 B] Get:5 http://raspbian.mirror.constant.com/raspbian bookworm/main armhf python3.11-venv armhf 3.11.2-6 [5,892 B] Fetched 3,043 kB in 2s (1,409 kB/s) Selecting previously unselected package python3-lib2to3. (Reading database ... 44873 files and directories currently installed.) Preparing to unpack .../0-python3-lib2to3_3.11.2-3_all.deb ... Unpacking python3-lib2to3 (3.11.2-3) ... Selecting previously unselected package python3-distutils. Preparing to unpack .../1-python3-distutils_3.11.2-3_all.deb ... Unpacking python3-distutils (3.11.2-3) ... Selecting previously unselected package python3-pip-whl. Preparing to unpack .../2-python3-pip-whl_23.0.1+dfsg-1_all.deb ... Unpacking python3-pip-whl (23.0.1+dfsg-1) ... Selecting previously unselected package python3-setuptools-whl. Preparing to unpack .../3-python3-setuptools-whl_66.1.1-1_all.deb ... Unpacking python3-setuptools-whl (66.1.1-1) ... Selecting previously unselected package python3.11-venv. Preparing to unpack .../4-python3.11-venv_3.11.2-6_armhf.deb ... Unpacking python3.11-venv (3.11.2-6) ... Selecting previously unselected package python3-venv. Preparing to unpack .../5-python3-venv_3.11.2-1_armhf.deb ... Unpacking python3-venv (3.11.2-1) ... Setting up python3-setuptools-whl (66.1.1-1) ... Setting up python3-pip-whl (23.0.1+dfsg-1) ... Setting up python3-lib2to3 (3.11.2-3) ... Setting up python3-distutils (3.11.2-3) ... Setting up python3.11-venv (3.11.2-6) ... Setting up python3-venv (3.11.2-1) ... root@raspberrypi:~# python3 -m venv /usr/local/ansible ``` 
root@raspberrypi:~# /usr/local/ansible/bin/python3 -m pip install --upgrade ansible-core
Collecting ansible-core
  Downloading ansible_core-2.15.3-py3-none-any.whl (2.2 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.2/2.2 MB 5.8 MB/s eta 0:00:00
Collecting jinja2>=3.0.0
  Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 133.1/133.1 kB 2.6 MB/s eta 0:00:00
Collecting PyYAML>=5.1
  Downloading PyYAML-6.0.1.tar.gz (125 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 125.2/125.2 kB 3.7 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cryptography
  Downloading cryptography-41.0.3.tar.gz (630 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 630.1/630.1 kB 6.7 MB/s eta 0:00:00
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [97 lines of output]
      Collecting setuptools>=61.0.0
        Using cached setuptools-68.2.0-py3-none-any.whl (807 kB)
      Collecting wheel
        Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
      Collecting cffi>=1.12
        Downloading cffi-1.15.1.tar.gz (508 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 508.5/508.5 kB 4.2 MB/s eta 0:00:00
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'done'
      Collecting setuptools-rust>=0.11.4
        Downloading setuptools_rust-1.7.0-py3-none-any.whl (25 kB)
      Collecting pycparser
        Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
           ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 1.8 MB/s eta 0:00:00
      Collecting semantic-version<3,>=2.8.2
        Downloading semantic_version-2.10.0-py2.py3-none-any.whl (15 kB)
      Collecting typing-extensions>=3.7.4.3
        Downloading typing_extensions-4.7.1-py3-none-any.whl (33 kB)
      Installing collected packages: wheel, typing-extensions, setuptools, semantic-version, pycparser, setuptools-rust, cffi
        DEPRECATION: cffi is being installed using the legacy 'setup.py install' method, because it does not have a 'pyproject.toml' and the 'wheel' package is not installed. pip 23.1 will enforce this behaviour change. A possible replacement is to enable the '--use-pep517' option. Discussion can be found at https://github.com/pypa/pip/issues/8559
        Running setup.py install for cffi: started
        Running setup.py install for cffi: finished with status 'error'
        error: subprocess-exited-with-error

        × Running setup.py install for cffi did not run successfully.
        │ exit code: 1
        ╰─> [60 lines of output]
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            Package libffi was not found in the pkg-config search path.
            Perhaps you should add the directory containing `libffi.pc'
            to the PKG_CONFIG_PATH environment variable
            Package 'libffi', required by 'virtual:world', not found
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/config/setupcfg.py:515: SetuptoolsDeprecationWarning: The license_file parameter is deprecated, use license_files instead.
              warnings.warn(msg, warning_class)
            running install
            /usr/local/ansible/lib/python3.11/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
              warnings.warn(
            running build
            running build_py
            creating build
            creating build/lib.linux-aarch64-cpython-311
            creating build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/error.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/__init__.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/commontypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/lock.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/pkgconfig.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/setuptools_ext.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/ffiplatform.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/backend_ctypes.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_gen.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/model.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/api.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/verifier.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cffi_opcode.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/recompiler.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/vengine_cpy.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/cparser.py -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_include.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/parse_c_type.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_embedding.h -> build/lib.linux-aarch64-cpython-311/cffi
            copying cffi/_cffi_errors.h -> build/lib.linux-aarch64-cpython-311/cffi
            running build_ext
            building '_cffi_backend' extension
            creating build/temp.linux-aarch64-cpython-311
            creating build/temp.linux-aarch64-cpython-311/c
            arm-linux-gnueabihf-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -DFFI_BUILDING=1 -DUSE__THREAD -DHAVE_SYNC_SYNCHRONIZE -I/usr/include/ffi -I/usr/include/libffi -I/usr/local/ansible/include -I/usr/include/python3.11 -c c/_cffi_backend.c -o build/temp.linux-aarch64-cpython-311/c/_cffi_backend.o
            c/_cffi_backend.c:2:10: fatal error: Python.h: No such file or directory
                2 | #include <Python.h>
                  |          ^~~~~~~~~~
            compilation terminated.
            error: command '/usr/bin/arm-linux-gnueabihf-gcc' failed with exit code 1
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: legacy-install-failure

      × Encountered error while trying to install package.
      ╰─> cffi

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for output from the failure.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Related:

bennuttall commented 9 months ago

@michaing any ideas?

MichaIng commented 9 months ago

Are you sure that you added piwheels as extra index? I see only the architecture independent "any" wheels being downloaded (and pip does not show whether from piwheels or PyPI), while all architecture dependent modules are downloaded as source archives and tried to be compiled in place. This of course fails if no development headers are installed, but is unrelated to piwheels.

If piwheels indeed has been added as index and other wheels are indeed downloaded form piwheels (which we cannot see here), then we'd need to check why cffi, cryptography and PyYAML are not.

holta commented 9 months ago

@MichaIng @bennuttall thanks for your help.

Just LMK if any additional testing results are needed. 💯

MichaIng commented 9 months ago

As said, currently it looks like you did not add piwheels as additional index. Please check your pip.conf and in case show some pip output where wheels are really downloaded from piwheels. Otherwise I see no issue.

bennuttall commented 9 months ago

Ugh sorry @MichaIng didn't read the output properly

holta commented 9 months ago

1) Any idea why /etc/pip.conf exists on 64-bit Raspberry Pi OS but not on 32-bit Raspberry Pi OS? Directly in opposition to what is claimed at the top of the main page https://piwheels.org here:

image

2) Great news is that adding the above file ( /etc/pip.conf ) to "32-bit" (running a 64-bit kernel) Raspberry Pi OS 12 / Bookworm pre-release is indeed confirmed to work, even/also in the venv. Thank you @MichaIng for explaining!

MichaIng commented 9 months ago

AFAIK, pip,conf is not present on any vanilla Raspberry Pi OS but always needs to be added manually. And yes, on the 64-bit variant it does not make any sense of course as piwheels does not currently support 64-bit.

EDIT: Ah, I just see the note that it is assumed to be present now. I wasn't aware of this, but makes sense (on the 32-bit variant). Strange is that I cannot see this being done neither in pi-gen, not as part of the raspberrypi-sys-mods package 🤔.

bennuttall commented 9 months ago

Actually /etc/pip.conf is added by the python-pip-whl package in pios. I'm not sure if it's included in aarch64 images, or in bookworm. Will make sure RPTL are aware it should be there. Even if we don't have aarch64 wheels yet, it does no harm and prepares for future support.

MichaIng commented 9 months ago

Ah I see: https://archive.raspberrypi.org/debian/pool/main/p/python-pip/

I thought that it might be overwritten by a newer version on Bookworm, but actually that package does not exist on Debian Bookworm at all: https://packages.debian.org/python-pip-whl Instead: https://packages.debian.org/python3-pip-whl

Now I see that python-pip-whl was a dependency of python-pip as well as python3-pip until Bullseye. On RPi OS lite this is not pre-installed, so if you do not install it manually, or instead install pip via get-pip.py, you won't have this package. Since Bookworm however, python-pip-whl has been removed (along with all Python 2 packages) and python3-pip-whl has not been added as dependency. So on a distro upgrade Bullseye => Bookworm python-pip-whl gets auto-removed and python3-pip-whl not installed. As RPi provides the python3-pip package from their repo, along with the adjusted python-pip-whl package, they'd need to add python3-pip-whl and add is as dependency to python3-pip for Bookworm builds of these packages to maintain the current behaviour.

bennuttall commented 9 months ago

Ah that makes sense. I'll let them know.

holta commented 9 months ago

python3-pip-whl has not been added as dependency. So on a distro upgrade Bullseye => Bookworm python-pip-whl gets auto-removed and python3-pip-whl not installed.

In Addition:

apt package python3-pip-whl was in fact installed — as a result of my manually running apt install python3-venv (as seen in original above readout).

Consequently RasPiOS may want to consider altering apt package python3-pip-whl — so that it too installs /etc/pip.conf :pray:

root@raspberrypi:~# apt install python3-venv
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following additional packages will be installed:
  python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3.11-venv
The following NEW packages will be installed:
  python3-distutils python3-lib2to3 python3-pip-whl python3-setuptools-whl python3-venv python3.11-venv
...
MichaIng commented 9 months ago

Reading the package description, the whole purpose of it is to be able to run pip within venv, so it totally makes sense to have it as dependency of python3-venv instead of python3-pip: https://packages.debian.org/bookworm/python3-pip-whl

Now that Debian ships the /usr/lib/python3.11/EXTERNALLY-MANAGED blocker, pip is generally seen to be used within venv only. And while pip is also often installed via https://bootstrap.pypa.io/get-pip.py, Python itself and venv are installed via Debian APT packages in most cases. So it is probably best to add pip.conf in python3-pip-whl and leave it as python3-venv dependency, so that it is automatically installed in the most common/intended cases.