Question: Universal2 wheel with binary extension

[thorstenhater]

Description

Hi,

after failing multiple times and finding no advice wrt towards fixing this on my own, I wonder whether someone here could help. We are packaging a native extension via scikit-build wrapped via pybind11 into wheels using cibuildwheel. Since a few weeks ago, our GHA CI job to produce wheels started failing on MacOS. The problem occurs during the repair/delocate phase, see build log.

Since the original error indicated a problem with a static library, see here https://github.com/arbor-sim/arbor/actions/runs/9345982063/job/25720046459 I forced CIBuildWheel / CMake to produce a shared library, which results in the current state of affairs.

Any hints towards fixing this? Thanks in advance.

Build log

https://github.com/arbor-sim/arbor/actions/runs/9346194861/job/25720548893?pr=2274

CI config

https://github.com/arbor-sim/arbor/pull/2274/files#diff-91dc372617f128032762d652aa093c427eb3b25dcbfd86998db90e3914ec7e79

[Czaki]

Did you have an artifact with static version of the library? It will be nice to understand why macholib cannot parse it.

Current log suggests that libarbor.dylib is not added to wheel by build system and path where it is stored is not provided to delocate.

[thorstenhater]

For static build, the stack trace is my only hint, but trying to search for the specific problem yielding nothing actionable. For reference:

  ValueError: Unknown Mach-O header: 0x213c6172 in <_io.BufferedReader name='/private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/tmpydaqd7gl/wheel/arbor/lib/libarbor.a'>

[Czaki]

This information is not enough. The best thing will be to start build with static library, but with disabled repair command, and then there will be an option to download artifact and check what happen.

[thorstenhater]

Sure, here's a version I produced with static linking and repairing is replaced by cp https://github.com/arbor-sim/arbor/actions/runs/9354636187/artifacts/1563733032 from job https://github.com/arbor-sim/arbor/actions/runs/9354636187/job/25747931818?pr=2274

[Czaki]

Thanks. First try to do not show the source of the problem. I will need to investigate more time.

In the context of the current problem, maybe @henryiii will have an idea what may be wrong with cmake configuration.

[thorstenhater]

Interestingly enough running the steps by hand locally, outside of cibuildwheel, works without any issues on an M1 Mac. That narrows down the obvious candidates for root causes a bit. I think. My test Mac is still on Ventura (13) and GHA seems to be running Sonoma (14). The other difference is that cibuildwheel produces an universal2 wheel and local builds have arm64 only. I will try to dig a bit deeper, too.

[Czaki]

You may specify macos-13 runner on CI to test this idea.

[thorstenhater]

Thanks for the tip, but I saw it a bit late and went the other route by testing using just arch=arm64 first. Which seems to be Ok, see here: https://github.com/arbor-sim/arbor/actions/runs/9366259599/job/25783299447?pr=2274 Salient changes:

[tool.cibuildwheel.macos]
#archs = ["universal2"]
archs = ["arm64"]
environment = { MACOSX_DEPLOYMENT_TARGET = "12.0.1" }
# repair-wheel-command = "cp {wheel} {dest_dir}"

Update: setting architectures to ["arm64", "x86_64"] is fine, too, and runs on MacOS 14. Thus I think universal2 is problematic for some reason.

For me, this is a workable solution. If you want to close this issue, it's fine for me, but I'll also help with finding the actual root cause.