Issues with libcompression.dylib and delocate 0.9

[gijzelaerr]

I suddenly start experiencing issues running delocate building our binary wheels, it looks like delocate is trying to modify system libraries and fails to do so.

delocate version 0.9 only, downgrading to 0.8.2 seems to resolve the issue. python version 3.6, 3.7, 3.8, 3.9 (pyenv) on OS X 10.15 (github actions).

output:

Run DYLD_LIBRARY_PATH=/usr/local/lib venv/bin/delocate-wheel -v dist/*.whl
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libmonetdbe.2.dylib to .dylibs/libmonetdbe.2.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libmonetdbsql.11.dylib to .dylibs/libmonetdbsql.11.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libmonetdb5.31.dylib to .dylibs/libmonetdb5.31.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libmapi.13.dylib to .dylibs/libmapi.13.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libbat.23.dylib to .dylibs/libbat.23.dylib
INFO:delocate.delocating:Copying library /usr/local/Cellar/monetdb/11.40.1/lib/libstream.15.dylib to .dylibs/libstream.15.dylib
INFO:delocate.delocating:Copying library /usr/local/lib/liblzma.5.dylib to .dylibs/liblzma.5.dylib
INFO:delocate.delocating:Copying library /usr/local/lib/liblz4.1.dylib to .dylibs/liblz4.1.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdbe.2.dylib from @rpath/libmonetdb5.31.dylib to @loader_path/libmonetdb5.31.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdbsql.11.dylib from @rpath/libmonetdb5.31.dylib to @loader_path/libmonetdb5.31.dylib
INFO:delocate.delocating:Modifying install name in _lowlevel.abi3.so from @rpath/libmonetdbe.2.dylib to @loader_path/.dylibs/libmonetdbe.2.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdbe.2.dylib from @rpath/libmonetdbsql.11.dylib to @loader_path/libmonetdbsql.11.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdbe.2.dylib from @rpath/libstream.15.dylib to @loader_path/libstream.15.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdbsql.11.dylib from @rpath/libstream.15.dylib to @loader_path/libstream.15.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmonetdb5.31.dylib from @rpath/libstream.15.dylib to @loader_path/libstream.15.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libmapi.13.dylib from @rpath/libstream.15.dylib to @loader_path/libstream.15.dylib
INFO:delocate.delocating:Modifying install name in .dylibs/libbat.23.dylib from @rpath/libstream.15.dylib to @loader_path/libstream.15.dylib
INFO:delocate.delocating:Modifying install name in ../../../../../../../../../usr/lib/libcompression.dylib from /usr/lib/liblzma.5.dylib to @loader_path/../../private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpgrokdy0q/wheel/monetdbe/.dylibs/liblzma.5.dylib
Fixing: dist/monetdbe-0.10.1-cp36-cp36m-macosx_10_14_x86_64.whl
Traceback (most recent call last):
  File "venv/bin/delocate-wheel", line 11, in <module>
    load_entry_point('delocate==0.9.0', 'console_scripts', 'delocate-wheel')()
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/cmd/delocate_wheel.py", line 99, in main
    ignore_missing=opts.ignore_missing_dependencies,
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/delocating.py", line 553, in delocate_wheel
    ignore_missing=ignore_missing,
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/delocating.py", line 444, in delocate_path
    return delocate_tree_libs(lib_dict, lib_path, tree_path)
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/delocating.py", line 90, in delocate_tree_libs
    lib_dict, root_path, libraries_to_delocate | copied_libraries
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/delocating.py", line 193, in _update_install_names
    set_install_name(requiring, orig_install_name, new_install_name)
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/tools.py", line 110, in modify
    return f(filename, *args, **kwargs)
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/tools.py", line 280, in set_install_name
    back_tick(['install_name_tool', '-change', oldname, newname, filename])
  File "/Users/runner/work/MonetDBe-Python/MonetDBe-Python/venv/lib/python3.6/site-packages/delocate/tools.py", line 60, in back_tick
    cmd_str, retcode, err.decode('latin-1')))
RuntimeError: install_name_tool -change /usr/lib/liblzma.5.dylib @loader_path/../../private/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/tmpgrokdy0q/wheel/monetdbe/.dylibs/liblzma.5.dylib /usr/lib/libcompression.dylib returned code 1 with error /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /usr/lib/libcompression.dylib (for architecture x86_64)
/Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: warning: changes being made to the file will invalidate the code signature in: /usr/lib/libcompression.dylib (for architecture x86_64h)
error: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: can't create output file: /usr/lib/libcompression.dylib.XXXXXX (Operation not permitted)
fatal error: /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: cannot rename /usr/lib/libcompression.dylib to /usr/lib/libcompression.dylib.XXXXXX (No such file or directory)

full build can be found here:

https://github.com/MonetDBSolutions/MonetDBe-Python/runs/3201098936?check_suite_focus=true

[matthew-brett]

@HexDecimal - any ideas here? I guess this is the refactor?

[gijzelaerr]

downgrading to 0.8.2 seems to resolve the issue.

[HexDecimal]

@HexDecimal - any ideas here? I guess this is the refactor?

The recursive dependency resolver noticed that the system library /usr/lib/libcompression.dylib had a reference back to the local library /usr/local/lib/liblzma.5.dylib and tried to update its install names. I'm guessing the local library took priority over a system library during dynamic linking.

This might have happened because of the differences in how lib_filt_func and copy_filt_func work. The code could be updated so that checks to lib_filt_func are always False whenever copy_filt_func is False.

[matthew-brett]

Aha - thanks for that analysis. I guess this is, in fact, a genuine error, in that it is not possible, in this case, to recreate the behavior of the code on the system, without copying the system lib.

[HexDecimal]

I can add a fix to #118 or add a fix in a separate PR.

Depending on the rpaths of system libraries it might be possible to create a test to check this.

[HexDecimal]

I think the best option would be to have lib_filt_func and copy_filt_func not be defined as separate parameters.

[HexDecimal]

Or maybe you can keep them separate to define intent, but combining them internally will resolve this issue.

[matthew-brett]

That was partly my point about this being a genuine error - in that the copy_filt_func says not to copy, but we discover that this means we can't delocate the package properly.

[matthew-brett]

I guess my question here is - should we raise a more informative error here? Or just pass silently by?

[HexDecimal]

I'm preparing a fix with #120. There won't be an error, files which aren't copied will no longer be followed. So Delocate will no longer have non-copied libraries depending on copied libraries.

[gijzelaerr]

thanks!