Closed matham closed 2 years ago
Thanks for the report - is there any way you can provide me the input wheel, so I can try to reproduce? Which xcode image are you using?
Here's the wheel under the py_wheel_osx artifact. I disabled the delocate
part of the action so it would upload the originals so there's no error in this one, but the error occurred under python 3.6 (the first wheel compiled in the python versions).
I'm not sure about xcode. I'm not very familiar with osx. But it's using macos-latest
, which is macOS Catalina 10.15
accoriding to github. Here's the relevant part of the recipe:
osx_wheels:
runs-on: macos-latest
steps:
- uses: actions/checkout@v1
- name: Brew setup
run: |
brew update
brew install pkg-config
brew install sdl2
brew install sdl2_mixer
brew install ffmpeg
pkg-config --libs --cflags libavcodec
- name: Building wheels
run: |
mkdir ~/ffpyplayer_wheels
for pyver in $OSX_PYVERS; do
git reset --hard
git clean -d -x -f
pyver_short=${pyver:0:3}
curl -O -L https://www.python.org/ftp/python/$pyver/python-$pyver-macosx10.9.pkg
sudo installer -package python-$pyver-macosx10.9.pkg -target /
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python$pyver_short get-pip.py --user
python$pyver_short -m pip install --upgrade --user pip
python$pyver_short -m pip install --upgrade --user cython pytest wheel
python$pyver_short -m pip install --upgrade delocate
python$pyver_short setup.py bdist_wheel
ls dist/
cp dist/*.whl ~/ffpyplayer_wheels
done
If you can, please try the code at #79 . I think it will fix this...
That indeed fixes it. Thanks for fixing it so quickly!
Hi, I am using delocate==0.10.0
on macOS 12.0 Beta 9
with Python 3.10 (installed via macOS 64-bit universal2 installer
) and I am encountering the same issue as reported above:
$ delocate-listdeps <name>-<version>-cp310-cp310-macosx_10_9_universal2.whl
Traceback (most recent call last):
File ".../.venv/bin/delocate-listdeps", line 8, in <module>
sys.exit(main())
File ".../.venv/lib/python3.10/site-packages/delocate/cmd/delocate_listdeps.py", line 44, in main
lib_dict = wheel_libs(path, ignore_missing=True)
File ".../.venv/lib/python3.10/site-packages/delocate/libsana.py", line 731, in wheel_libs
lib_dict = tree_libs_from_directory(
File ".../.venv/lib/python3.10/site-packages/delocate/libsana.py", line 375, in tree_libs_from_directory
return _tree_libs_from_libraries(
File ".../.venv/lib/python3.10/site-packages/delocate/libsana.py", line 300, in _tree_libs_from_libraries
for depending_path, install_name in get_dependencies(
File ".../.venv/lib/python3.10/site-packages/delocate/libsana.py", line 100, in get_dependencies
for install_name in get_install_names(lib_fname):
File ".../.venv/lib/python3.10/site-packages/delocate/tools.py", line 246, in get_install_names
names = tuple(parse_install_name(line)[0] for line in lines[1:])
File ".../.venv/lib/python3.10/site-packages/delocate/tools.py", line 246, in <genexpr>
names = tuple(parse_install_name(line)[0] for line in lines[1:])
File ".../.venv/lib/python3.10/site-packages/delocate/tools.py", line 163, in parse_install_name
return IN_RE.match(line).groups()
AttributeError: 'NoneType' object has no attribute 'groups'
$ xcode-select -p
/Library/Developer/CommandLineTools
$ otool --version
llvm-otool(1): Apple Inc. version cctools-986
otool(1): Apple Inc. version cctools-986
disassmbler: LLVM version 13.0.0, (clang-1300.0.29.3)
Edit: I can reproduce the issue using Python 3.8 (pre-installed) on macOS Big Sur 11.6
. In both cases, I am using a M1 MacBook.
parse_install_name
should really handle what happens when the match fails.
@papr, you should post your wheel if you can.
This is the wheel: ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl.zip
I wasn't able to reproduce the error with Xcode 13 on macOS 11.
@papr can you test the #130 branch and provide an updated traceback?
❯ delocate-listdeps --version
delocate-listdeps 0.10.0+15.g8077870
❯ delocate-listdeps ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
Traceback (most recent call last):
File "/Users/ppr/work/pupil/.venv/bin/delocate-listdeps", line 8, in <module>
sys.exit(main())
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/cmd/delocate_listdeps.py", line 55, in main
lib_dict = wheel_libs(path, ignore_missing=True)
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/libsana.py", line 749, in wheel_libs
lib_dict = tree_libs_from_directory(
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/libsana.py", line 384, in tree_libs_from_directory
return _tree_libs_from_libraries(
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/libsana.py", line 309, in _tree_libs_from_libraries
for depending_path, install_name in get_dependencies(
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/libsana.py", line 105, in get_dependencies
for install_name in get_install_names(lib_fname):
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/tools.py", line 256, in get_install_names
names = tuple(parse_install_name(line)[0] for line in lines[1:])
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/tools.py", line 256, in <genexpr>
names = tuple(parse_install_name(line)[0] for line in lines[1:])
File "/Users/ppr/work/pupil/.venv/lib/python3.10/site-packages/delocate/tools.py", line 170, in parse_install_name
raise ValueError(f"Could not parse {line!r}")
ValueError: Could not parse '/private/var/folders/25/24p4_ynx17qf5p5ghpf9x7sr0000gn/T/tmpmnpne_qf/ndsi/writer.cpython-310-darwin.so (architecture arm64):'
❯ unzip ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
Archive: ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
inflating: ndsi/__init__.py
inflating: ndsi/formatter.py
inflating: ndsi/frame.cpython-310-darwin.so
inflating: ndsi/network.py
inflating: ndsi/sensor.py
inflating: ndsi/writer.cpython-310-darwin.so
inflating: ndsi-1.4.2.dist-info/LICENSE
inflating: ndsi-1.4.2.dist-info/METADATA
inflating: ndsi-1.4.2.dist-info/WHEEL
inflating: ndsi-1.4.2.dist-info/top_level.txt
inflating: ndsi-1.4.2.dist-info/RECORD
❯ otool -L ndsi/writer.cpython-310-darwin.so
ndsi/writer.cpython-310-darwin.so (architecture x86_64):
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
ndsi/writer.cpython-310-darwin.so (architecture arm64):
/opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib (compatibility version 0.0.0, current version 0.2.0)
/opt/homebrew/opt/ffmpeg/lib/libavutil.56.dylib (compatibility version 56.0.0, current version 56.70.100)
/opt/homebrew/opt/ffmpeg/lib/libavformat.58.dylib (compatibility version 58.0.0, current version 58.76.100)
/opt/homebrew/opt/ffmpeg/lib/libavcodec.58.dylib (compatibility version 58.0.0, current version 58.134.100)
/opt/homebrew/opt/ffmpeg/lib/libswscale.5.dylib (compatibility version 5.0.0, current version 5.9.100)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
That's perfect, showing the otool output makes it clear what happened.
This code here removes only the first line which is supposed to be the "architecture" line and then doesn't expect there to be a 2nd architecture. This seems to cause the current error.
Right now I don't know what the correct way of handling this is supposed to be. I'd guess ignoring the separate architectures and merging the sets of libraries might work. I wasn't able to reproduce the error so I can't write a test for this.
I did not figure out how to build a proper universal wheel yet... but once I do, I expect the architecture x86_64
entries to point to the same file names but at different locations. The x86_64 homebrew uses a different prefix than the arm64 one.
As a result, delocating everything into .dylibs/
might cause name clashes. Maybe it makes to delocate the entries for each architecture into a dedicated subfolder, i.e. .dylibs/x86_64/
and .dylibs/arm64/
?
I think I've fixed the immediate issue in PR #130 which can now parse and track install names with multiple architectures. @papr, I'd like you to test it in case any other otool parsers are affected, and maybe show the output of otool -D
if that becomes relevant.
I looked at Homebrew and it seems their reasoning is that they've tried but their setup can't be changed over to universal binaries so easily. I'd suggest making separate wheels for each architecture until Homebrew ever has an official solution, since the other options are going to be difficult to implement and easy to mess up unless you have both architectures to test on.
Bundling multiple architectures is beyond my abilities right now. If you're serious about that suggestion that then I'd like you to open a separate issue for it so it can be tracked.
Yes, building wheels for each architecture separately is probably the easiest way.
❯ unzip ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl.zip
Archive: ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl.zip
inflating: ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
❯ delocate-listdeps --version
delocate-listdeps 0.10.0+18.g57a0457
❯ delocate-listdeps ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.10/bin/delocate-listdeps", line 8, in <module>
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/cmd/delocate_listdeps.py", line 55, in main
lib_dict = wheel_libs(path, ignore_missing=True)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/libsana.py", line 749, in wheel_libs
lib_dict = tree_libs_from_directory(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/libsana.py", line 384, in tree_libs_from_directory
return _tree_libs_from_libraries(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/libsana.py", line 309, in _tree_libs_from_libraries
for depending_path, install_name in get_dependencies(
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/libsana.py", line 105, in get_dependencies
for install_name in get_install_names(lib_fname):
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/delocate/tools.py", line 350, in get_install_names
assert names_for_arch[0] == install_id
AssertionError
❯ unzip ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
Archive: ndsi-1.4.2-cp310-cp310-macosx_10_9_universal2.whl
inflating: ndsi/__init__.py
inflating: ndsi/formatter.py
inflating: ndsi/frame.cpython-310-darwin.so
inflating: ndsi/network.py
inflating: ndsi/sensor.py
inflating: ndsi/writer.cpython-310-darwin.so
inflating: ndsi-1.4.2.dist-info/LICENSE
inflating: ndsi-1.4.2.dist-info/METADATA
inflating: ndsi-1.4.2.dist-info/WHEEL
inflating: ndsi-1.4.2.dist-info/top_level.txt
inflating: ndsi-1.4.2.dist-info/RECORD
❯ otool -L ndsi/writer.cpython-310-darwin.so
ndsi/writer.cpython-310-darwin.so (architecture x86_64):
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
ndsi/writer.cpython-310-darwin.so (architecture arm64):
/opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib (compatibility version 0.0.0, current version 0.2.0)
/opt/homebrew/opt/ffmpeg/lib/libavutil.56.dylib (compatibility version 56.0.0, current version 56.70.100)
/opt/homebrew/opt/ffmpeg/lib/libavformat.58.dylib (compatibility version 58.0.0, current version 58.76.100)
/opt/homebrew/opt/ffmpeg/lib/libavcodec.58.dylib (compatibility version 58.0.0, current version 58.134.100)
/opt/homebrew/opt/ffmpeg/lib/libswscale.5.dylib (compatibility version 5.0.0, current version 5.9.100)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
❯ otool -D ndsi/writer.cpython-310-darwin.so
ndsi/writer.cpython-310-darwin.so (architecture x86_64):
ndsi/writer.cpython-310-darwin.so (architecture arm64):
❯ otool -D ndsi/*.so
ndsi/frame.cpython-310-darwin.so (architecture x86_64):
ndsi/frame.cpython-310-darwin.so (architecture arm64):
ndsi/writer.cpython-310-darwin.so (architecture x86_64):
ndsi/writer.cpython-310-darwin.so (architecture arm64):
I printed the variables of the assert to see what is going on:
AssertionError: install_id='/private/var/folders/25/24p4_ynx17qf5p5ghpf9x7sr0000gn/T/tmpimk0avs3/ndsi/writer.cpython-310-darwin.so (architecture arm64):' names_for_arch=['/usr/lib/libc++.1.dylib', '/usr/lib/libSystem.B.dylib']
You can always build wheels for the two architectures, delocate them, and then fuse them into one wheel with the delocate-fuse
command.
I missed delocate-fuse
and lipo
. Automatically merging architectures might be easier than I originally thought, but I'd hesitate to do that without knowing about the situations where lipo
could fail or cause problems.
I think I have install name ids fixed in #130. The changes in otool's output broke everything since it all was hard coded to expect only the one section instead of the possibility of a section per-architecture. Luckily it's only parsed the 3 places and I've fixed two to them so far, thanks to papr showing the output since I don't have access to that myself.
The last one is the rpaths parser using otool -l
. I'd like @papr to show an example output again, and if you don't have libraries with rpaths then you can use this wheel.
Apologies for the delayed response. Somehow this fall off my radar.
Successfully installed delocate-0.10.0+19.g06a0fa8.dirty
❯ delocate-listdeps /Users/ppr/Downloads/fakepkg_rpath-1.0-cp36-abi3-macosx_10_9_universal2.whl
@rpath/libextfunc_rpath.dylib not found:
Needed by: /private/var/folders/25/24p4_ynx17qf5p5ghpf9x7sr0000gn/T/tmp863r_gu1/fakepkg/subpkg/module2.abi3.so
Search path:
libs/
@rpath/libextfunc_rpath.dylib not found:
Needed by: /private/var/folders/25/24p4_ynx17qf5p5ghpf9x7sr0000gn/T/tmp863r_gu1/fakepkg/subpkg/module2.abi3.so
Search path:
libs/
@rpath/libextfunc_rpath.dylib not found, requested by /private/var/folders/25/24p4_ynx17qf5p5ghpf9x7sr0000gn/T/tmp863r_gu1/fakepkg/subpkg/module2.abi3.so
❯ otool -l ~/Downloads/fakepkg_rpath-1.0-cp36-abi3-macosx_10_9_universal2/fakepkg/subpkg/module2.abi3.so | gh gist create
- Creating gist...
✓ Created gist
❯ otool -L ndsi/writer.cpython-310-darwin.so
ndsi/writer.cpython-310-darwin.so (architecture x86_64):
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
ndsi/writer.cpython-310-darwin.so (architecture arm64):
/opt/homebrew/opt/jpeg-turbo/lib/libturbojpeg.0.dylib (compatibility version 0.0.0, current version 0.2.0)
/opt/homebrew/opt/ffmpeg/lib/libavutil.56.dylib (compatibility version 56.0.0, current version 56.70.100)
/opt/homebrew/opt/ffmpeg/lib/libavformat.58.dylib (compatibility version 58.0.0, current version 58.76.100)
/opt/homebrew/opt/ffmpeg/lib/libavcodec.58.dylib (compatibility version 58.0.0, current version 58.134.100)
/opt/homebrew/opt/ffmpeg/lib/libswscale.5.dylib (compatibility version 5.0.0, current version 5.9.100)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 905.6.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
❯ otool -l ndsi/writer.cpython-310-darwin.so | gh gist create
- Creating gist...
✓ Created gist
https://gist.github.com/3099b913b07a38de758b121c97cb1984
❯ delocate-listdeps ndsi-1.4.3-cp310-cp310-macosx_10_9_universal2.whl
/opt/homebrew/Cellar/aom/3.1.3/lib/libaom.3.1.3.dylib
/opt/homebrew/Cellar/dav1d/0.9.2/lib/libdav1d.5.dylib
/opt/homebrew/Cellar/ffmpeg/4.4_2/lib/libavcodec.58.134.100.dylib
/opt/homebrew/Cellar/ffmpeg/4.4_2/lib/libavformat.58.76.100.dylib
/opt/homebrew/Cellar/ffmpeg/4.4_2/lib/libavutil.56.70.100.dylib
/opt/homebrew/Cellar/ffmpeg/4.4_2/lib/libswresample.3.9.100.dylib
/opt/homebrew/Cellar/ffmpeg/4.4_2/lib/libswscale.5.9.100.dylib
/opt/homebrew/Cellar/fontconfig/2.13.1/lib/libfontconfig.1.dylib
/opt/homebrew/Cellar/freetype/2.11.0/lib/libfreetype.6.dylib
/opt/homebrew/Cellar/gettext/0.21/lib/libintl.8.dylib
/opt/homebrew/Cellar/gmp/6.2.1_1/lib/libgmp.10.dylib
/opt/homebrew/Cellar/gnutls/3.6.16_1/lib/libgnutls.30.dylib
/opt/homebrew/Cellar/jpeg-turbo/2.1.1/lib/libturbojpeg.0.2.0.dylib
/opt/homebrew/Cellar/lame/3.100/lib/libmp3lame.0.dylib
/opt/homebrew/Cellar/libbluray/1.3.0/lib/libbluray.2.dylib
/opt/homebrew/Cellar/libffi/3.4.2/lib/libffi.8.dylib
/opt/homebrew/Cellar/libidn2/2.3.2/lib/libidn2.0.dylib
/opt/homebrew/Cellar/libogg/1.3.5/lib/libogg.0.dylib
/opt/homebrew/Cellar/libpng/1.6.37/lib/libpng16.16.dylib
/opt/homebrew/Cellar/libsodium/1.0.18_1/lib/libsodium.23.dylib
/opt/homebrew/Cellar/libsoxr/0.1.3/lib/libsoxr.0.1.2.dylib
/opt/homebrew/Cellar/libtasn1/4.17.0/lib/libtasn1.6.dylib
/opt/homebrew/Cellar/libunistring/0.9.10/lib/libunistring.2.dylib
/opt/homebrew/Cellar/libvorbis/1.3.7/lib/libvorbis.0.dylib
/opt/homebrew/Cellar/libvorbis/1.3.7/lib/libvorbisenc.2.dylib
/opt/homebrew/Cellar/nettle/3.7.3/lib/libhogweed.6.4.dylib
/opt/homebrew/Cellar/nettle/3.7.3/lib/libnettle.8.4.dylib
/opt/homebrew/Cellar/opencore-amr/0.1.5/lib/libopencore-amrnb.0.dylib
/opt/homebrew/Cellar/opencore-amr/0.1.5/lib/libopencore-amrwb.0.dylib
/opt/homebrew/Cellar/openjpeg/2.4.0/lib/libopenjp2.2.4.0.dylib
/opt/homebrew/Cellar/openssl@1.1/1.1.1l_1/lib/libcrypto.1.1.dylib
/opt/homebrew/Cellar/openssl@1.1/1.1.1l_1/lib/libssl.1.1.dylib
/opt/homebrew/Cellar/opus/1.3.1/lib/libopus.0.dylib
/opt/homebrew/Cellar/p11-kit/0.24.0_1/lib/libp11-kit.0.dylib
/opt/homebrew/Cellar/rav1e/0.4.1_1/lib/librav1e.0.4.1.dylib
/opt/homebrew/Cellar/snappy/1.1.9/lib/libsnappy.1.1.9.dylib
/opt/homebrew/Cellar/speex/1.2.0/lib/libspeex.1.dylib
/opt/homebrew/Cellar/srt/1.4.4/lib/libsrt.1.4.4.dylib
/opt/homebrew/Cellar/theora/1.1.1/lib/libtheoradec.1.dylib
/opt/homebrew/Cellar/theora/1.1.1/lib/libtheoraenc.1.dylib
/opt/homebrew/Cellar/webp/1.2.1/lib/libwebp.7.dylib
/opt/homebrew/Cellar/webp/1.2.1/lib/libwebpmux.3.dylib
/opt/homebrew/Cellar/x264/r3060/lib/libx264.163.dylib
/opt/homebrew/Cellar/x265/3.5/lib/libx265.199.dylib
/opt/homebrew/Cellar/xz/5.2.5/lib/liblzma.5.dylib
/opt/homebrew/Cellar/zeromq/4.3.4/lib/libzmq.5.dylib
❯ delocate-listdeps --version
delocate-listdeps 0.10.0+19.g06a0fa8.dirty
diff --git a/delocate/tools.py b/delocate/tools.py
index 16464f1..43fdcae 100644
--- a/delocate/tools.py
+++ b/delocate/tools.py
@@ -390,7 +390,7 @@ def get_install_names(filename: str) -> Tuple[str, ...]:
names_for_arch = [name for name, _, _ in libraries]
if arch in install_ids:
# Remove redundant install id from the install names.
- assert names_for_arch[0] == install_ids[arch]
+ assert names_for_arch[0] == install_ids[arch], f"{install_ids[arch]=} {names_for_arch=}"
names_for_arch = names_for_arch[1:]
for name in names_for_arch:
if name not in all_names: # Avoid duplicates and preserve order.
hey whats the status of this? is there an update?
currently it still seems like the best solution is to not build universal binaries and build it separately :/
I have PR #130 but I wanted it to be reviewed before merging it.
You can use pip install git+https://github.com/HexDecimal/delocate.git@fix-78
to test this PR yourself.
well it works in my case :) thx
I'm running delocate on github actions on a osx build and it errors out with the following error:
The command I'm running is
delocate-wheel dist/*.whl
, and the result ofls dist/
printsffpyplayer-4.3.2.dev0-cp36-cp36m-macosx_10_9_x86_64.whl
indicating that the file is there.