search

pyca / cryptography

cryptography is a package designed to expose cryptographic primitives and recipes to Python developers.
https://cryptography.io
Other
6.6k stars 1.51k forks source link

Cannot install cryptography on Fedora 41+, CentOS Stream 10 #11690

Open CtrlZmaster opened 1 day ago

CtrlZmaster commented 1 day ago

Basically the same issue as #11331, result of OpenSSL engine deprecation. Fedora 41 is in beta now and CentOS Stream 10 is starting to be added to various CI tools, so I think this issue is getting more impactful.

Pip will build from source after specifying --no-binary or if wheel for an architecture is not available. Upstream cryptography is not installable by pip for s390x and ppc64le architectures on these distributions at all. There was #11328 which was closed without merging even though it looked like it worked and there were no objections. Could it be merged, please?

Fedora 41

[root@f41-a ~]# pip install --verbose --no-binary :all: cryptography
Using pip 24.2 from /usr/lib/python3.13/site-packages/pip (python 3.13)
Collecting cryptography
  Using cached cryptography-43.0.1.tar.gz (686 kB)
  Running command pip subprocess to install build dependencies
  Using pip 24.2 from /usr/lib/python3.13/site-packages/pip (python 3.13)
  Collecting maturin<2,>=1
    Using cached maturin-1.7.4-cp313-cp313-linux_x86_64.whl
  Collecting cffi>=1.12
    Using cached cffi-1.17.1-cp313-cp313-linux_x86_64.whl
  Collecting setuptools!=74.0.0,!=74.1.0,!=74.1.1
    Using cached setuptools-75.1.0-py3-none-any.whl
  Collecting pycparser (from cffi>=1.12)
    Using cached pycparser-2.22-py3-none-any.whl
  Installing collected packages: setuptools, pycparser, maturin, cffi
  Successfully installed cffi-1.17.1 maturin-1.7.4 pycparser-2.22 setuptools-75.1.0
  WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  Getting requirements to build wheel ... done
  Running command Preparing metadata (pyproject.toml)
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE"
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE.APACHE"
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE.BSD"
  🍹 Building a mixed python/rust project
  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
  🐍 Not using a specific python interpreter
  📡 Using build options features, locked from pyproject.toml
  cryptography-43.0.1.dist-info
  Checking for Rust toolchain....
  Running `maturin pep517 write-dist-info --metadata-directory /tmp/pip-modern-metadata-nw3quv22 --interpreter /usr/bin/python3`
  Preparing metadata (pyproject.toml) ... done
Collecting cffi>=1.12 (from cryptography)
  Using cached cffi-1.17.1-cp313-cp313-linux_x86_64.whl
Collecting pycparser (from cffi>=1.12->cryptography)
  Using cached pycparser-2.22-py3-none-any.whl
Building wheels for collected packages: cryptography
  Running command Building wheel for cryptography (pyproject.toml)
  Running `maturin pep517 build-wheel -i /usr/bin/python3 --compatibility off`
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE"
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE.APACHE"
  📦 Including license file "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/LICENSE.BSD"
  🍹 Building a mixed python/rust project
  🔗 Found pyo3 bindings with abi3 support for Python ≥ 3.7
  🐍 Not using a specific python interpreter
  📡 Using build options features, locked from pyproject.toml
     Compiling target-lexicon v0.12.15
     Compiling proc-macro2 v1.0.86
     Compiling unicode-ident v1.0.12
     Compiling cc v1.1.6
     Compiling pyo3-build-config v0.22.2
     Compiling quote v1.0.36
     Compiling syn v2.0.71
     Compiling pkg-config v0.3.30
     Compiling vcpkg v0.2.15
     Compiling once_cell v1.19.0
     Compiling openssl-sys v0.9.103
     Compiling libc v0.2.155
     Compiling cfg-if v1.0.0
     Compiling pyo3-macros-backend v0.22.2
     Compiling pyo3-ffi v0.22.2
     Compiling autocfg v1.3.0
     Compiling memoffset v0.9.1
     Compiling openssl v0.10.66
     Compiling foreign-types-shared v0.1.1
     Compiling heck v0.5.0
     Compiling foreign-types v0.3.2
     Compiling pyo3 v0.22.2
     Compiling bitflags v2.6.0
     Compiling unindent v0.2.3
     Compiling indoc v2.0.5
     Compiling cryptography-key-parsing v0.1.0 (/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/cryptography-key-parsing)
     Compiling cryptography-cffi v0.1.0 (/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/cryptography-cffi)
     Compiling cryptography-openssl v0.1.0 (/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/cryptography-openssl)
  The following warnings were emitted during compilation:

  warning: cryptography-cffi@0.1.0: /tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out/_openssl.c:638:10: fatal error: openssl/engine.h: No such file or directory
  warning: cryptography-cffi@0.1.0:   638 | #include <openssl/engine.h>
  warning: cryptography-cffi@0.1.0:       |          ^~~~~~~~~~~~~~~~~~
  warning: cryptography-cffi@0.1.0: compilation terminated.

  error: failed to run custom build command for `cryptography-cffi v0.1.0 (/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/cryptography-cffi)`

  Caused by:
    process didn't exit successfully: `/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-6a99f89988fe3b02/build-script-build` (exit status: 1)
    --- stdout
    cargo:rustc-check-cfg=cfg(python_implementation, values("CPython", "PyPy"))
    cargo:rerun-if-env-changed=PYO3_PYTHON
    cargo:rerun-if-changed=../../_cffi_src/
    cargo:rerun-if-changed=../../cryptography/__about__.py
    cargo:rustc-cfg=python_implementation="CPython"
    OUT_DIR = Some(/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out)
    TARGET = Some(x86_64-unknown-linux-gnu)
    OPT_LEVEL = Some(3)
    HOST = Some(x86_64-unknown-linux-gnu)
    cargo:rerun-if-env-changed=CC_x86_64-unknown-linux-gnu
    CC_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CC_x86_64_unknown_linux_gnu
    CC_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CC
    HOST_CC = None
    cargo:rerun-if-env-changed=CC
    CC = None
    cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
    RUSTC_WRAPPER = None
    cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
    CRATE_CC_NO_DEFAULTS = None
    DEBUG = Some(false)
    CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
    cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
    CFLAGS_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
    CFLAGS_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CFLAGS
    HOST_CFLAGS = None
    cargo:rerun-if-env-changed=CFLAGS
    CFLAGS = None
    OUT_DIR = Some(/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out)
    cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
    cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
    cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
    CFLAGS_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
    CFLAGS_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CFLAGS
    HOST_CFLAGS = None
    cargo:rerun-if-env-changed=CFLAGS
    CFLAGS = None
    OUT_DIR = Some(/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out)
    cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
    cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
    cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
    CFLAGS_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
    CFLAGS_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CFLAGS
    HOST_CFLAGS = None
    cargo:rerun-if-env-changed=CFLAGS
    CFLAGS = None
    OUT_DIR = Some(/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out)
    cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
    cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
    cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
    CFLAGS_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
    CFLAGS_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CFLAGS
    HOST_CFLAGS = None
    cargo:rerun-if-env-changed=CFLAGS
    CFLAGS = None
    OUT_DIR = Some(/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out)
    cargo:rerun-if-env-changed=CC_ENABLE_DEBUG_OUTPUT
    cargo:rerun-if-env-changed=CRATE_CC_NO_DEFAULTS
    CRATE_CC_NO_DEFAULTS = None
    CARGO_CFG_TARGET_FEATURE = Some(fxsr,sse,sse2)
    cargo:rerun-if-env-changed=CFLAGS_x86_64-unknown-linux-gnu
    CFLAGS_x86_64-unknown-linux-gnu = None
    cargo:rerun-if-env-changed=CFLAGS_x86_64_unknown_linux_gnu
    CFLAGS_x86_64_unknown_linux_gnu = None
    cargo:rerun-if-env-changed=HOST_CFLAGS
    HOST_CFLAGS = None
    cargo:rerun-if-env-changed=CFLAGS
    CFLAGS = None
    cargo:warning=/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out/_openssl.c:638:10: fatal error: openssl/engine.h: No such file or directory
    cargo:warning=  638 | #include <openssl/engine.h>
    cargo:warning=      |          ^~~~~~~~~~~~~~~~~~
    cargo:warning=compilation terminated.

    --- stderr

    error occurred: Command "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-m64" "-I" "/usr/include" "-I" "/usr/include/python3.13" "-Wall" "-Wextra" "-Wconversion" "-Wno-error=sign-conversion" "-Wno-unused-parameter" "-fmacro-prefix-map=/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out=." "-DPy_LIMITED_API=0x030700f0" "-o" "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out/3b3415f8e17501e8-_openssl.o" "-c" "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/target/release/build/cryptography-cffi-aa5ee4323e7684cb/out/_openssl.c" with args cc did not execute successfully (status code exit status: 1).

  warning: build failed, waiting for other jobs to finish...
  💥 maturin failed
    Caused by: Failed to build a native library through cargo
    Caused by: Cargo build finished with "exit status: 101": `env -u CARGO PYO3_ENVIRONMENT_SIGNATURE="cpython-3.13-64bit" PYO3_PYTHON="/usr/bin/python3" PYTHON_SYS_EXECUTABLE="/usr/bin/python3" "cargo" "rustc" "--features" "pyo3/abi3-py37" "--message-format" "json-render-diagnostics" "--locked" "--manifest-path" "/tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc/src/rust/Cargo.toml" "--release" "--lib"`
  Error: command ['maturin', 'pep517', 'build-wheel', '-i', '/usr/bin/python3', '--compatibility', 'off'] returned non-zero exit status 1
  error: subprocess-exited-with-error

  × Building wheel for cryptography (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /usr/bin/python3 /usr/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /tmp/tmpfe0e2w5w
  cwd: /tmp/pip-install-efxjzd2r/cryptography_4757ed757203406189be02ed0ab66efc
  Building wheel for cryptography (pyproject.toml) ... error
  ERROR: Failed building wheel for cryptography

CentOS Stream 10

alex commented 1 day ago

That PR was closed without merging because we had follow up questions before it could be merged and never got a response.

Those questions remain, namely: Why RedHat's behavior here is different from upstream's no-engine, which we support and test against.

If we can get a clear answer to that question, we can proceed, but from my perspective this is blocked on Red Hat, not us.

CtrlZmaster commented 1 day ago

Ah, my bad, I read the discussion wrong in #11328.

I do not feel fully qualified to answer, I am just a cryptography user, so by no means an expert on OpenSSL. But I read the change proposal for Fedora and the discussion for a rejected proposal to remove OpenSSL engines. I think that I see what is going on here and I understand this step from maintenance/packaging perspective.

First, there were concerns that providers still have issues and prevent full switchover from engines. That prompted another approach (the second accepted proposal), deprecating engines but still keeping them for packages that cannot switch over to providers. So OpenSSL is not built with --no-engine. Instead, the engine headers are simply moved from package openssl-devel to openssl-devel-engines which is marked as deprecated. And then OPENSSL_NO_ENGINE is defined mimicking the build without engine support. Fedora Packaging Guidelines prevent additions of new packages with deprecated dependencies. Packages that are ready to switch can do it now. Then there are existing packages that might not know about the deprecation of engines or cannot switch to providers yet. This approach will give them time to coordinate the switch after their builds will start to fail (this has already happened). So, they can easily fix for now by depending on openssl-devel-engines and start replacing them.

I would also like to point out that this is not a "Red Hat behavior" or a Red Hat decision. This approach was chosen by Fedora community, during a Fedora Change process and approved by FESCo (Fedora Engineering Steering Committee). CentOS Stream 10 is simply following the Fedora approach as its downstream.