Missing library files at repair step & wheel configuration through pyproject

marchelbling-aqemia commented 3 months ago

Description

First of all, thank you for providing cibw to the community!

Missing library files at repair step

I'm trying to provide a pre-built linux only wheel for OpenBabel. I'm taking inspiration from https://github.com/njzjz/openbabel-wheel which is unfortunately not maintained anymore.

The basic setup is that the repository I build only has a CMakeLists.txt that will be used by cibw/scikit-build-core to build the wheel:

CMakeLists.txt

``` cmake_minimum_required(VERSION 3.11) project(ObabelPythonDistributions CXX) # download OpenBabel to ${OBABEL_SOURCE_ROOT} include(FetchContent) FetchContent_Declare(openbabel GIT_REPOSITORY https://github.com/openbabel/openbabel GIT_TAG ${OBABEL_VERSION} ) FetchContent_GetProperties(openbabel) if(NOT openbabel_POPULATED) FetchContent_Populate(openbabel) add_subdirectory(${openbabel_SOURCE_DIR} ${openbabel_BINARY_DIR}) install( DIRECTORY ${openbabel_SOURCE_DIR}/scripts/python/openbabel DESTINATION ${CMAKE_INSTALL_PREFIX}/openbabel ) endif() ```

My pyproject defines the version and build flags I want to use:

pyproject.toml

``` [tool.scikit-build] minimum-version = "0.3.0" [tool.scikit-build.cmake.define] OBABEL_VERSION = "f495cc6" CMAKE_BUILD_TYPE = "Release" WITH_INCHI = "ON" PYTHON_BINDINGS = "ON" RUN_SWIG = "ON" [tool.cibuildwheel] archs = ["x86_64"] build = ["cp311-manylinux2014-x86_64"] test-command = [ 'python -c "from openbabel import openbabel"', "obabel -:C -oxyz --gen3d", ] [tool.cibuildwheel.linux] before-all = [ "yum install -y zlib-devel-1.2.7 eigen3-devel-3.3.7 libxml2-devel-2.9.1", ] ```

and I use Github actions to build everything.

The issue I have is that the wheel builds fine:

truncated install logs

``` 2024-08-23T08:19:42.0620747Z *** Installing project into wheel... 2024-08-23T08:19:42.1426718Z -- Install configuration: "Release" 2024-08-23T08:19:42.1427901Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel 2024-08-23T08:19:42.1437400Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/openbabel/openbabel/pybel.py ... 2024-08-23T08:19:42.2599648Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0 2024-08-23T08:19:42.2607617Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7 2024-08-23T08:19:42.2614782Z -- Set non-toolchain portion of runtime path of "/tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so.7.0.0" to "/tmp/tmpmvydrfap/wheel/platlib/openbabel-aqemia/lib" 2024-08-23T08:19:42.2621652Z -- Installing: /tmp/tmpmvydrfap/wheel/platlib/lib/libopenbabel.so ... 2024-08-23T08:19:42.4828912Z *** Making wheel... 2024-08-23T08:19:45.5498324Z *** Created aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl 2024-08-23T08:19:45.5688505Z Building wheel for aqemia-openbabel (pyproject.toml): finished with status 'done' 2024-08-23T08:19:45.5786590Z Created wheel for aqemia-openbabel: filename=aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl size=11102455 sha256=363d1d05dad22183022cd9e4f7c9f8a62d912bc44394f8835c03ef171b461551 2024-08-23T08:19:45.5788988Z Stored in directory: /root/.cache/pip/wheels/21/14/79/baf708a1463c156ec37e5504ed91f66de9ca8b1bc844f71781 2024-08-23T08:19:45.5824955Z Successfully built aqemia-openbabel 2024-08-23T08:19:45.5940993Z Removed build tracker: '/tmp/pip-build-tracker-q2u7ftms' 2024-08-23T08:19:45.6790642Z + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)' 2024-08-23T08:19:45.7065003Z + rm -rf /tmp/cibuildwheel/repaired_wheel 2024-08-23T08:19:45.7091788Z + mkdir -p /tmp/cibuildwheel/repaired_wheel 2024-08-23T08:19:45.7124575Z ##[endgroup] ```

however the auditwheel repair fails as it cannot find libraries since they only live in the wheel file

logs

``` 2024-08-23T08:19:45.7126351Z ##[group]Repairing wheel... 2024-08-23T08:19:45.7126696Z 2024-08-23T08:19:45.7130107Z + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl' 2024-08-23T08:19:46.0501385Z INFO:auditwheel.main_repair:Repairing aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl 2024-08-23T08:20:06.1838904Z Traceback (most recent call last): 2024-08-23T08:20:06.1870673Z File "/usr/local/bin/auditwheel", line 8, in 2024-08-23T08:20:06.1871685Z sys.exit(main()) 2024-08-23T08:20:06.1873139Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main.py", line 54, in main 2024-08-23T08:20:06.1874430Z rval = args.func(args, p) 2024-08-23T08:20:06.1875837Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/main_repair.py", line 173, in execute 2024-08-23T08:20:06.1877130Z out_wheel = repair_wheel( 2024-08-23T08:20:06.1878534Z File "/opt/_internal/pipx/venvs/auditwheel/lib/python3.10/site-packages/auditwheel/repair.py", line 78, in repair_wheel 2024-08-23T08:20:06.1879819Z raise ValueError( 2024-08-23T08:20:06.1880749Z ValueError: Cannot repair wheel, because required library "libopenbabel.so.7" could not be located 2024-08-23T08:20:06.5140592Z ##[endgroup] 2024-08-23T08:20:06.5147019Z [31m✕ [0m20.80s 2024-08-23T08:20:06.5184462Z ##[error]Command ['sh', '-c', 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/aqemia_openbabel-0.0.1-cp311-cp311-linux_x86_64.whl'] failed with code 1. ```

I believe I must be missing something obvious here but I don't see how to fix the problem except by using a before-all action that basically builds the project. This doubles the build time which is not great. Would there be other options that I missed?

Wheel configuration through pyproject

Also I'm struggling defining the wheels I want to build. I'm only interested in building cp311-manylinux2014-x86_64. When I set my pyproject.toml (which is what I understand as the preferred way) to

 [tool.cibuildwheel]
 archs = ["x86_64"]
 build = ["cp311-manylinux2014-x86_64"]
 platform = ["linux"]

it seems that cibuildwheel does't pick the configuration and the action run fails with

cibuildwheel: No build identifiers selected: BuildSelector(build_config='cp311-manylinux2014-x86_64', skip_config='', requires_python=<SpecifierSet('<3.12,>=3.11')>, prerelease_pythons=False, free_threaded_support=False)

So I end up defining CIBW_ARCHS, CIWB_BUILD and CIWB_PLATFORM which I would rather not. Is there some precedence rule that I misunderstood?

Thanks a lot for any help!

Build log

No response

CI config

No response

Czaki commented 3 months ago

It looks like you should use build = ["cp311-manylinux2014_x86_64"]

henryiii commented 3 months ago

The build identifiers don’t include the manylinux docker image variant (like 2014). build = ["cp311-manylinux_x86_64"]

Czaki commented 3 months ago

I miss this update. How to specify now the manylinux image to use?

joerick commented 3 months ago

See the table in the docs for a full list of build identifiers, they don't include the manylinux variant as Henry says.

I miss this update. How to specify now the manylinux image to use?

You can choose the image you want to use with the manylinux-ARCH-image options. It's always been this way 🙃

henryiii commented 3 months ago

Do you have a link to the repo? I'd start by running pipx run build --wheel and seeing what is actually in the wheel.

marchelbling-aqemia commented 3 months ago

Thanks a lot for your answers! Sorry, I did not pay attention to the fact that the variant was not part of the build parameter. This matter is all resolved. I assumed that part of the log (build_config='cp311-manylinux2014-x86_64') was actually confirming that this wasn't an issue in providing the right values

For the "double build", here is a repo reproducing my setup: https://github.com/marchelbling/wheels/. Note that I actually realized that the Python dev dependencies were not picked by cmake and the python extension was not built. I updated the CMakeLists.txt but it seems that I still have issues at link time.

After re-reading the scikit-build-core documentation:

I'm starting to think that I will not manage to build OpenBabel using the manylinux image as I understand it does not include Python libraries. Am I correct?

henryiii commented 3 months ago

Wheels are not supposed to "embed" Python. Python is already present when are you loading a wheel, so you don't need to link to it, and it could even lock you to the patch release you linked against. So yes, the Python libs are removed when building the manylinux image. See https://github.com/pypa/manylinux/pull/1185 for some context.

marchelbling-aqemia commented 3 months ago

Thank you @henryiii that really helped me! It would have been awesome that the PR was merged but it was not too difficult to apply the PR diff on manylinux main branch.

Now my wheel finally builds cleanly except for the "double" build though. Note that I found another project that seems to have the same issue: https://github.com/spglib/spglib/blob/d9c572d1986b0eccc4ed5f18c8d827729a320313/pyproject.toml#L111-L112. Please let me know if you would have a chance to look at what I could be doing wrong.

marchelbling-aqemia commented 2 months ago

I actually came across https://github.com/pypa/auditwheel/issues/285. I'm not sure I fully understand the conclusion from the thread. I'm really not sure what the "proper" way to fix this. I stopped building twice and I'm now doing a manual repair where I unzip the wheel and set LD_LIBRARY_PATH before running auditwheel repair but that doesn't seem "right".

pypa / cibuildwheel