Non-python extension dependencies with external dependencies are not recursively patched

[ehashman]

As reported by @thomaslima:

Our wheel has a complicated dependency structure. But the gist of it is the following:

dbcore.so (py_ext) -> lib_tl.so (non-py_ext) -> libcurl.so (external library to be grafted)

Because lib_tl is a non-py extension of dbcore, it was not added to the full_external_refs dictionary, which is used by repair.py to patch the sonames. So the final result is that lib_tl was not patched, and the package was broken.

See also #134.

[dsteinmo]

Is there confirmation that the PR proposed by @thomaslima fixes this problem? My project also has a complicated dependency tree, and it would be ideal if all external dependencies that are bundled get patched to include $ORIGIN in rpath. Thanks.

[grisuthedragon]

We have the same problem that our python extension c_interface.so depends on a library libcode.so which we also ship in the .libs folder. This this is require since the build process of libcode.so does not depend on python and we only create the c_interface.so using setuptools. libcode.so has all dependencies to the external libaries inside.

@dsteinmo With the PR #139 integrated in the auditwheel script, the problem is fixed and the wheels work well.

[dsteinmo]

Sounds good. It does not appear that @thomaslima has engaged the upstream maintainers by rebasing his branch. If I find some time, I'll attempt a fork + new PR.

[grisuthedragon]

I extracted the crucual part out of @thomaslima commit, which we use to build our wheels.

Here is the patch: auditwheel.txt

[dsteinmo]

Thanks; I might have spoken too soon, it looks like this is still active, but in a separate branch alongside a separate issue: https://github.com/pypa/auditwheel/pull/134. I'll leave it in the hands of @thomaslima and the maintainers.

[thomaslima]

Dear all,

I’ve been very busy in the past weeks. I will try to look into this soon. Originally I had one single PR and Elana asked me to split into two. That might have created some confusion. I’ll look into it again about what’s required.

Best, Thomas

On Aug 29, 2019, at 08:39, Derek Steinmoeller notifications@github.com<mailto:notifications@github.com> wrote:

Thanks; I might have spoken too soon, it looks like this is still active, but in a separate branch alongside a separate issue: #134https://github.com/pypa/auditwheel/pull/134. I'll leave it in the hands of @thomaslimahttps://github.com/thomaslima and the maintainers.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/pypa/auditwheel/issues/136?email_source=notifications&email_token=AAIJZEZ6OL55RD2J2GAAAHDQG67RFA5CNFSM4GR7RLL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5OKSHA#issuecomment-526166300, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AAIJZE5R6QZ2O3FD6QNXMYLQG67RFANCNFSM4GR7RLLQ.

[delarosatrevin]

Hi everyone,

Is there any progress on this issue? I'm facing the same problem to build a wheel for my package emcore.

It is a scientific image processing library that depends on external libraries such as fftw3, libpng, libtiff, and sqlite3 among others. We build libemcore that does not depends on Python and then the Python extension using pybind11.

Thanks in advance.

[myselfhimself]

It seems that this command: (relying on the copydeps package)

copydeps gmic.cpython-37m-x86_64-linux-gnu.so --exclude exclude-list-gmic

with exclude-list-gmic's contents here

finds and copies shared libraries deeper than PR #139 ... Still struggling with auditwheel+ PR #139's shallow shared libraries lookup and copy into a repaired wheel archive... (libcurl)

[myselfhimself]

My copydeps-based script did not work well.

I am giving up on depending on libcurl for linux wheel building and will rely instead of automatic OS execution of the curl executable by the gmic library.

I am getting back to a standard auditwheel version

[bdice]

This problem is happening for me as well (see issue https://github.com/glotzerlab/freud/issues/720). I have confirmed that PR #139 does solve the problem for our use case. We have a few Python extensions that link to libfreud.so and libtbb.so. libfreud.so is also linked to libtbb.so. auditwheel correctly relocates the RPATHs of our Python extensions, but does not alter the RPATH of libfreud.so to point to the relocated libtbb.so.

It looks like PR #139 was stalled because the original author needed to update it. I will update it and file a new PR so that this can hopefully be fixed in the next release of auditwheel.

[thomaslima]

I’m very sorry I haven’t had the time to review the updates on the PR and issues. Thank you for working on this.

From: Bradley Dice notifications@github.com Date: Wednesday, February 3, 2021 at 16:44 To: pypa/auditwheel auditwheel@noreply.github.com Cc: Thomas Ferreira de Lima thomas.ferreira.de.lima@Princeton.EDU, Mention mention@noreply.github.com Subject: Re: [pypa/auditwheel] Non-python extension dependencies with external dependencies are not recursively patched (#136)

This problem is happening for me as well (see issue glotzerlab/freud#720https://github.com/glotzerlab/freud/issues/720). I have confirmed that PR #139https://github.com/pypa/auditwheel/pull/139 does solve the problem for our use case. We have a few Python extensions that link to libfreud.so and libtbb.so. libfreud.so is also linked to libtbb.so. auditwheel correctly relocates the RPATHs of our Python extensions, but does not relocate the RPATH of libfreud.so.

It looks like PR #139https://github.com/pypa/auditwheel/pull/139 was stalled because the original author needed to update it. I will update it and file a new PR so that this can hopefully be fixed in the next release of auditwheel.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/pypa/auditwheel/issues/136#issuecomment-772844107, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AAIJZE5SI2TMEZTMU6C2S2TS5G7PVANCNFSM4GR7RLLQ.

[bdice]

@thomaslima No problem! Thanks for your initial work on it. I am updating the tests, too. PR #134 with the corresponding tests was reverted in #190, so I will try to improve it according to the maintainers’ comments on why it was reverted.

[bdice]

@thomaslima I created a new pull request https://github.com/pypa/auditwheel/pull/283 to solve this issue, based on your previous work. Thanks for your initial efforts in #139, it just took a little bit of cleanup and updating tests! 👍 🚀

[molpopgen]

Am I mistaken, or is this a duplicate of #48? Came across both after running into this issue myself.