search

pypa / auditwheel

Auditing and relabeling cross-distribution Linux wheels.
Other
443 stars 145 forks source link

Making a manylinux2010 wheel with dynamically loaded libraries #197

Open gijzelaerr opened 5 years ago

gijzelaerr commented 5 years ago

hi! I'm a developer maintaining a python package that dynamically loads a library, the python-snap7 project to be specific. I got very excited when I learned it is now quite easy to make manylinux2010 wheels, but then I came accross this:

auditwheel uses the DT_NEEDED information (like ldd) from the Python extension modules to determine which system system libraries they depend on. Code that dynamically loads libraries at runtime using ctypes / cffi (from Python) or dlopen (from C/C++) doesn't contain this information in a way that can be statically determined, so dependencies that are loaded via those mechanisms will be missed.

And indeed my modules loaded by ctypes are not picked up. I've been googlin for a while now, but I can't find a good example or documentation or even it is possible to force the inclusion of a specific SO into the wheel. Is it impossibly hard to do?

Sorry to bother you if this is the wrong location to ask this question or address this issue, but for any python package maintainer loading libraries dynamically I think it could be extremely useful to have the documentation a bit more extended with a section on how to accomplish described issue (if that is even possible).

Thanks!

lkollar commented 4 years ago

Auditwheel could could be extended to take additional external libraries on the command line and vendor those in as well. I think the easiest would be to do this as an extra step after the main repair_wheel, or possibly even as a separate sub-command (e.g auditwheel vendor).

gijzelaerr commented 4 years ago

Hi @lkollar, thank you for your reply! Good to know that it is just not possible at the moment.

FirefoxMetzger commented 2 years ago

We, too, have a package that linkes to a library using ctypes. Calling auditwheel on the resulting wheel fails with INFO:auditwheel.main_repair:This does not look like a platform wheel, which is incorrect (it does include a platform-specific shared library).

Is there any known workaround for this?

gijzelaerr commented 2 years ago

we managed to work around the limitations in our repo https://github.com/gijzelaerr/python-snap7/blob/master/.github/workflows/build-and-test.yml

reynoldsnlp commented 1 year ago

Auditwheel could could be extended to take additional external libraries on the command line and vendor those in as well. I think the easiest would be to do this as an extra step after the main repair_wheel, or possibly even as a separate sub-command (e.g auditwheel vendor).

@lkollar I think it would be more intuitive and user-friendly to accept an argument like auditwheel repair --include-lib /path/to/my/lib.so, so everything happens in one step.

This is something I would be happy to work on if you think it's a good first issue.

gijzelaerr commented 1 year ago

FYI we created a workaround by just linking to the SO with a dummy C file https://github.com/gijzelaerr/python-snap7/blob/master/setup.py#L19

you might also find useful that we do all kinds of other magic to repack the resulting wheels to improve python arch independence (if I remember correctly):

https://github.com/gijzelaerr/python-snap7/blob/master/.github/workflows/build-and-test.yml https://github.com/gijzelaerr/python-snap7/blob/master/.github/build_scripts/repack_wheel.sh

edit: I only notice my own comment above now :) I'm getting old and repeating myself.