search

matthew-brett / delocate

Find and copy needed dynamic libraries into python wheels
BSD 2-Clause "Simplified" License
262 stars 59 forks source link

delocate-wheel sets invalid relative paths to delocated dylibs (related to "purelib" subdirectory) #149

Open maxhgerlach opened 2 years ago

maxhgerlach commented 2 years ago

Describe the bug / To Reproduce My original wheel contains these files:

$ unzip -l dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
Archive:  dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  4444243  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
      105  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/__init__.py
      215  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/METADATA
      108  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/WHEEL
       18  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/top_level.txt
      619  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/RECORD
---------                     -------
  4445308                     6 files

It depends on Boost, ICU, and PCRE, which have been installed via Homebrew:

$ delocate-listdeps dist/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_atomic-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_chrono-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_date_time-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_filesystem-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_iostreams-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_log-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_random-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_regex-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_serialization-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_system-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_thread-mt.dylib
/opt/homebrew/Cellar/boost/1.78.0_1/lib/libboost_unit_test_framework-mt.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicudata.70.1.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicui18n.70.1.dylib
/opt/homebrew/Cellar/icu4c/70.1/lib/libicuuc.70.1.dylib
/opt/homebrew/Cellar/pcre/8.45/lib/libpcre.1.dylib

I call delocate-wheel like this:

$ delocate-wheel -w dist-patched -v dist/*whl

The dylibs are properly copied into the delocated wheel:

$ unzip -l dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
Archive:  dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
  1040912  03-15-2022 12:31   PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib
    88048  03-15-2022 12:31   PyMyLib.dylibs/libboost_chrono-mt.dylib
   288224  03-15-2022 12:31   PyMyLib.dylibs/libpcre.1.dylib
   227408  03-15-2022 12:31   PyMyLib.dylibs/libboost_filesystem-mt.dylib
   180704  03-15-2022 12:31   PyMyLib.dylibs/libboost_thread-mt.dylib
    35456  03-15-2022 12:31   PyMyLib.dylibs/libboost_system-mt.dylib
  2780192  03-15-2022 12:31   PyMyLib.dylibs/libicui18n.70.1.dylib
    89968  03-15-2022 12:31   PyMyLib.dylibs/libboost_atomic-mt.dylib
   154208  03-15-2022 12:31   PyMyLib.dylibs/libboost_iostreams-mt.dylib
  1748992  03-15-2022 12:31   PyMyLib.dylibs/libicuuc.70.1.dylib
 29723472  03-15-2022 12:31   PyMyLib.dylibs/libicudata.70.1.dylib
  1337616  03-15-2022 12:31   PyMyLib.dylibs/libboost_log-mt.dylib
    84288  03-15-2022 12:31   PyMyLib.dylibs/libboost_random-mt.dylib
   449360  03-15-2022 12:31   PyMyLib.dylibs/libboost_regex-mt.dylib
    35536  03-15-2022 12:31   PyMyLib.dylibs/libboost_date_time-mt.dylib
   634416  03-15-2022 12:31   PyMyLib.dylibs/libboost_serialization-mt.dylib
      105  03-15-2022 11:19   PyMyLib-0.1.17.data/purelib/PyMyLib/__init__.py
  4462352  03-15-2022 12:31   PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
     2376  03-15-2022 12:31   PyMyLib-0.1.17.dist-info/RECORD
      108  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/WHEEL
       18  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/top_level.txt
      215  03-15-2022 11:19   PyMyLib-0.1.17.dist-info/METADATA
---------                     -------
 43363974                     22 files

The output of delocate-listdeps looks reasonable at first sight:

$ delocate-listdeps dist-patched/PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
PyMyLib.dylibs/libboost_atomic-mt.dylib
PyMyLib.dylibs/libboost_chrono-mt.dylib
PyMyLib.dylibs/libboost_date_time-mt.dylib
PyMyLib.dylibs/libboost_filesystem-mt.dylib
PyMyLib.dylibs/libboost_iostreams-mt.dylib
PyMyLib.dylibs/libboost_log-mt.dylib
PyMyLib.dylibs/libboost_random-mt.dylib
PyMyLib.dylibs/libboost_regex-mt.dylib
PyMyLib.dylibs/libboost_serialization-mt.dylib
PyMyLib.dylibs/libboost_system-mt.dylib
PyMyLib.dylibs/libboost_thread-mt.dylib
PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib
PyMyLib.dylibs/libicudata.70.1.dylib
PyMyLib.dylibs/libicui18n.70.1.dylib
PyMyLib.dylibs/libicuuc.70.1.dylib
PyMyLib.dylibs/libpcre.1.dylib

However, otool -L shows me these relative paths:

$ cd dist-patched/
dist-patched$ unzip PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
dist-patched$ otool -L PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so
PyMyLib-0.1.17.data/purelib/PyMyLib/PyMyLib.cpython-39-darwin.so:
    @loader_path/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_iostreams-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_random-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_serialization-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_system-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_unit_test_framework-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_log-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libicudata.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
    @loader_path/../../../PyMyLib.dylibs/libicui18n.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
    @loader_path/../../../PyMyLib.dylibs/libicuuc.70.1.dylib (compatibility version 70.0.0, current version 70.1.0)
    @loader_path/../../../PyMyLib.dylibs/libpcre.1.dylib (compatibility version 4.0.0, current version 4.13.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_chrono-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_regex-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_filesystem-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_thread-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    @loader_path/../../../PyMyLib.dylibs/libboost_atomic-mt.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1200.3.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1311.0.0)

Note all the relative paths @loader_path/../../../PyMyLib.dylibs. While these match the files as they are placed inside the wheel archive, after pip install these relative paths don't point to the dylibs under site-packages:

dist-patched$ pip install -U --force-reinstall PyMyLib-0.1.17-cp39-cp39-macosx_12_0_arm64.whl
# in my virtual environment:
.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages$ ls PyMyLib*
PyMyLib:
PyMyLib.cpython-39-darwin.so __init__.py                            __pycache__

PyMyLib-0.1.17.dist-info:
INSTALLER       METADATA        RECORD          REQUESTED       WHEEL           direct_url.json top_level.txt

PyMyLib.dylibs:
libboost_atomic-mt.dylib              libboost_iostreams-mt.dylib           libboost_serialization-mt.dylib       libicudata.70.1.dylib
libboost_chrono-mt.dylib              libboost_log-mt.dylib                 libboost_system-mt.dylib              libicui18n.70.1.dylib
libboost_date_time-mt.dylib           libboost_random-mt.dylib              libboost_thread-mt.dylib              libicuuc.70.1.dylib
libboost_filesystem-mt.dylib          libboost_regex-mt.dylib               libboost_unit_test_framework-mt.dylib libpcre.1.dylib

Note how PyMyLib.cpython-39-darwin.so and __init__.py have been installed directly to PyMyLib/, not to PyMyLib-0.1.17.data/purelib/PyMyLib/ (as it looks in the wheel archive).

Consequently, I get import errors because the dynamic libraries are not found:

$ python -c 'import PyMyLib'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/__init__.py", line 1, in <module>
    from .PyMyLib import PyMyLib as MyLib
ImportError: dlopen(...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/PyMyLib.cpython-39-darwin.so, 0x0002): Library not loaded: @loader_path/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib
  Referenced from: ...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/PyMyLib.cpython-39-darwin.so
  Reason: tried: '...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/PyMyLib/../../../PyMyLib.dylibs/libboost_date_time-mt.dylib' (no such file), '/usr/local/lib/libboost_date_time-mt.dylib' (no such file), '/usr/lib/libboost_date_time-mt.dylib' (no such file)

Expected behavior delocate-wheel should set relative paths to dylibs in such a way that they agree with their ultimate location after pip install.

Wheels used Unfortunately not open source. But the __init__.py contains just these lines:

from .PyMyLib import PyMyLib as MyLib

if __name__ == "__main__":
    pass

Platform (please complete the following information):

$ pip --version
pip 22.0.4 from ...../.venv-devel-macos-arm64-tf26/lib/python3.9/site-packages/pip (python 3.9)

Additional context I actually don't know why our (cmake-based) build process produces wheels with these subdirectories PyMyLib-0.1.17.data/purelib. Pointers that could help me simplify that would also be appreciated. 🙂

planetmarshall commented 2 months ago

I know this bug report is a bit old but came across this today whilst packaging an existing CMake project into a wheel. It hapens because delocate doesn't take account of the final locations of files installed into the "data" folder in the wheel, as these go directly into the root of the environment.