search

jvolkman / repairwheel

Repair any wheel, anywhere
MIT License
38 stars 3 forks source link

Repairwheel breaks on musllinux (via cibuildwheel) #43

Open PiotrCzapla opened 1 month ago

PiotrCzapla commented 1 month ago

Hi,

I've added the repairwheels step to setup.py and it is working fine. Unfrotunately your tool breaks when run via cibuildwheel on musllinux. I seems that you ship with audittool that only works for manylinux.

I've tested the audittool installed stand alone and it works as it ships with different set of platforms on musllinux, but it fails when run via repairwheel. Have a look, 

/project # AUDITWHEEL_PLAT='musllinux_1_2_x86_64'  /opt/python/cp38-cp38/bin/python -m auditwheel repair /tmp/pywhispercpp-1.2.0-cp38-cp38-linux_x86_64.whl 
INFO:auditwheel.main_repair:Repairing pywhispercpp-1.2.0-cp38-cp38-linux_x86_64.whl
INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
INFO:auditwheel.wheeltools:New filename tags: musllinux_1_2_x86_64
INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp38-cp38-linux_x86_64
INFO:auditwheel.wheeltools:New WHEEL info tags: cp38-cp38-musllinux_1_2_x86_64
INFO:auditwheel.main_repair:
Fixed-up wheel written to /project/wheelhouse/pywhispercpp-1.2.0-cp38-cp38-musllinux_1_2_x86_64.whl

But it breaks when run via repairwheel:

/project # AUDITWHEEL_PLAT='musllinux_1_2_x86_64'  /opt/python/cp38-cp38/bin/python -m repairwheel /tmp/pywhispercpp-1.2.0-cp38-cp38-linux_x86_64.whl 
usage: __main__.py [-h] -o OUTPUT_DIR [-l LIB_DIR] [--no-sys-paths] [-V] wheel
__main__.py: error: the following arguments are required: -o/--output-dir
/project # AUDITWHEEL_PLAT='musllinux_1_2_x86_64'  /opt/python/cp38-cp38/bin/python -m repairwheel /tmp/pywhispercpp-1.2.0-cp38-cp38-linux_x86_64.whl  -o /tmp
Traceback (most recent call last):
  File "/opt/python/cp38-cp38/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/python/cp38-cp38/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/python/cp38-cp38/lib/python3.8/site-packages/repairwheel/__main__.py", line 4, in <module>
    main()
  File "/opt/python/cp38-cp38/lib/python3.8/site-packages/repairwheel/repair.py", line 106, in main
    fn(original_wheel, temp_wheel_dir, lib_path, use_sys_paths)
  File "/opt/python/cp38-cp38/lib/python3.8/site-packages/repairwheel/linux/repair.py", line 50, in repair
    main_repair.configure_parser(repair_sub_parsers)
  File "/opt/python/cp38-cp38/lib/python3.8/site-packages/repairwheel/_vendor/auditwheel/main_repair.py", line 43, in configure_parser
    p.add_argument(
  File "/opt/python/cp38-cp38/lib/python3.8/argparse.py", line 1368, in add_argument
    action = action_class(**kwargs)
  File "/opt/python/cp38-cp38/lib/python3.8/site-packages/repairwheel/_vendor/auditwheel/tools.py", line 122, in __init__
    raise argparse.ArgumentError(self, msg % args)
argparse.ArgumentError: argument --plat: invalid choice: 'musllinux_1_2_x86_64' from environment variable 'AUDITWHEEL_PLAT' (choose from 'linux_x86_64', 'manylinux_2_5_x86_64', 'manylinux_2_12_x86_64', 'manylinux_2_17_x86_64', 'manylinux_2_24_x86_64', 'manylinux_2_27_x86_64', 'manylinux_2_28_x86_64', 'manylinux_2_31_x86_64', 'manylinux_2_34_x86_64', 'manylinux_2_35_x86_64', 'manylinux1_x86_64', 'manylinux2010_x86_64', 'manylinux2014_x86_64')

to replicate this issue use docker 'musllinux_1_2_x86_64' docker run -it quay.io/pypa/musllinux_1_2_x86_64:2024.10.07-1 /bin/sh

Then pip install repair tool: /opt/python/cp38-cp38/bin/python -m pip install repairwheel

and try to run repairwheel with AUDITWHEEL_PLAT='musllinux_1_2_x86_64'

agriyakhetarpal commented 1 month ago

I remembered that there was some discussion about this over at https://github.com/ziglang/zig-pypi/issues/20#issuecomment-2088748050. Opening an issue here slipped out of my mind back then, so, thanks for this, @PiotrCzapla!

agriyakhetarpal commented 1 month ago

One (hacky) way to provide cross-platform support for this could be to ship[^1] the MUSL object files libc.musl-x86_64.so.1, libc.musl-aarch64.so.1, and those for other Linux architectures) within the repairwheel wheels as package data, and make them available when trying to repair a MUSL wheel.

This is on similar lines to what Zig does for its highly coveted cross-compilation support across platforms/architectures – as a white lie, it ships a copy of whatever libc is required to compile for a potential target: https://andrewkelley.me/post/zig-cc-powerful-drop-in-replacement-gcc-clang.html

[^1]: I'm not sure where they can be sourced from, though, since I wasn't able to find pre-compiled distributions for them so far.

PiotrCzapla commented 1 month ago

Thank you for prompt replay. While cross-compilation is appealing, it adds another layer of complexity, and we'll already be dealing with enough challenges when compiling with CUDA, Vulkan, support. For the time being we utilize repairwheel only during the pip install . process when it is not running inside of cibuildwheel where we relay on integration with auditwheel.

jvolkman commented 1 month ago

I originally built this tool to use with cross compilation workflows, so that's a big requirement.

The issue with musl is that - if I understand correctly - there's no way to tell that a binary is linked to musl libc - or which version of musl libc - just by looking at it. auditwheel just assumes that if you're running repair on a system with musl libc installed, any binary you're repairing is a musl binary. This is also not correct when cross compiling, but works in more standard usage.

Supporting musl should be possible - I just don't know of a way to do so automatically while also supporting the cross compilation workflow. Maybe a CLI param or environment variable that specifies the musl version.