dylibbundler is missing dependencies

jjeising commented 7 years ago

I would like to fix a binary with some more complex GTK dependencies. This kind of works, but the dependencies of pulled in dylibs sometimes get lost. Example:

The binary depends on libatkmm and libatk:

$ otool -L [binary] | grep atk
    /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/opt/atk/lib/libatk-1.0.0.dylib (compatibility version 22210.0.0, current version 22210.1.0)

dylibbundler has fixed this link and copied libatkmm-1.6.1.dylib. It also pulled in libatk (the original C library) and fixed it's dependencies. But the dependency on libatk of libaktmm is not fixed:

$ otool -L libatk-1.0.0.dylib
libatk-1.0.0.dylib:
    @executable_path/../libs/libatk-1.0.0.dylib (compatibility version 22210.0.0, current version 22210.1.0)
    @executable_path/../libs/libgobject-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.0.0)
    @executable_path/../libs/libglib-2.0.0.dylib (compatibility version 5001.0.0, current version
…
$ otool -L libatkmm-1.6.1.dylib
libatkmm-1.6.1.dylib:
    @executable_path/../libs/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/lib/libatk-1.0.0.dylib (compatibility version 21810.0.0, current version 21810.1.0)
    @executable_path/../libs/libglibmm-2.4.1.dylib (compatibility version 5.0.0, current version 5.0.0)
    @executable_path/../libs/libgobject-2.0.0.dylib (compatibility version 4601.0.0, current version 4601.2.0)
…

The commit of @chearon seems to fix this – the build works with this version.

auriamg commented 7 years ago

Hi, I'm no longer actively maintaining dylibbundler, but if someone wants to make a pull request I would be more than happy to merge it. Unfortunately chearon's work might be mergeable as-is since the commit message reads "this should NOT be relied on for new work because I didn't really take the time to fully understand how it's suppose to work"

chearon commented 7 years ago

Yeah, I saw how I could get it to work in a quick way and didn't go any deeper. I think I see where the problem really is now, though, I'll open a PR soon

chearon commented 7 years ago

@jjeising is there anything else depending on the C library libatk before you bundle? or just libatkmm?

jjeising commented 7 years ago

@chearon The binary itself depends on libatk, but with a different path. The dependencies are a little complex (38 in total) so I've just picked a simple one. But I should have checked this more thorough.

I have updated the post above to reflect this. You can also see the different path for libatk here:

otool -L /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib | grep atk
/usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib:
    /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/lib/libatk-1.0.0.dylib (compatibility version 21810.0.0, current version 21810.1.0)

(vs. /usr/local/opt/atk/lib/libatk-1.0.0.dylib in the binary).

Symlinks are at different levels:

/usr/local/opt/atk -> ../Cellar/atk/2.22.0
/usr/local/lib/libatk-1.0.0.dylib -> ../Cellar/atk/2.22.0/lib/libatk-1.0.0.dylib

chearon commented 7 years ago

Thanks, I ask because it actually turns out that my situation is a little different than the one I described in my commit. I think Homebrew had multiple copies of the same library, which my patch fixed because it patches ALL paths with the same base filename.

Just in case your situation is mine, and not a symlink issue, what does:

readlink /usr/local/opt/atk/lib/libatk-1.0.0.dylib readlink /usr/local/lib/libatk-1.0.0.dylib

output?

jjeising commented 7 years ago

$ readlink /usr/local/opt/atk/lib/libatk-1.0.0.dylib
$ readlink /usr/local/lib/libatk-1.0.0.dylib
../Cellar/atk/2.22.0/lib/libatk-1.0.0.dylib

$ realpath /usr/local/opt/atk/lib/libatk-1.0.0.dylib
/usr/local/Cellar/atk/2.22.0/lib/libatk-1.0.0.dylib
$ realpath /usr/local/lib/libatk-1.0.0.dylib
/usr/local/Cellar/atk/2.22.0/lib/libatk-1.0.0.dylib

chearon commented 7 years ago

those readlinks indicate that the first is a real file, and the second is a symbolic link to a different file. I'd think this would be an issue with Homebrew

jjeising commented 7 years ago

It is a real file because /usr/local/opt/atk is linked to the current version in the cellar:

/usr/local/opt/atk -> ../Cellar/atk/2.22.0

This is expected behaviour from homebrew as far as I know. This path is used for formulas that are keg-only (not linked to /usr/local/lib. Why mixed pathes in a single lib, I don't know.

But as the realpath of both libs points to the same target shouldn't dylibbundler replace both?

chearon commented 7 years ago

My mistake, you're right, the issue is multiple symbolic links pointing to the same file. Which is also my problem. The code uses the same prefix for all symbolic links (it expects them all to be in the same directory) which I think is the issue. If it stored the full path instead then all symbolic links would be correctly patched, is there a reason they are expected to be in the same directory @auriamg ?

auriamg commented 7 years ago

dylibbundler may have limitations simply because I never encountered any such case, I don't think there's any fundamental reason why this couldn't be supported

jjeising commented 7 years ago

@chearon I've tested your pull request #19, it looks like it's still missing something (commenting here for references):

These are fine:

$ otool -L [new-binary] | grep atk
    @executable_path/../libs/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    @executable_path/../libs/libatk-1.0.0.dylib (compatibility version 22210.0.0, current version 22210.1.0)
$otool -L .../libs/libatk-1.0.0.dylib
.../libs/libatk-1.0.0.dylib:
    @executable_path/../libs/libatk-1.0.0.dylib (compatibility version 22210.0.0, current version 22210.1.0)
    @executable_path/../libs/libgobject-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.0.0)
    @executable_path/../libs/libglib-2.0.0.dylib (compatibility version 5001.0.0, current version 5001.0.0)
    @executable_path/../libs/libintl.8.dylib (compatibility version 10.0.0, current version 10.5.0)
...

But one dependency is untouched:

otool -L .../libs/libatkmm-1.6.1.dylib | grep atk
.../libs/libatkmm-1.6.1.dylib:
    @executable_path/../libs/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/lib/libatk-1.0.0.dylib (compatibility version 21810.0.0, current version 21810.1.0)

Your old patch didn't show this behaviour. I hope I got the path from the pull request right, but I double checked and it looks like the right version.

chearon commented 7 years ago

@jjeising did you do make clean && make? I made a test program that links against atkmm and it seems to work for me: http://pastebin.ubuntu.com/23447534/

Edit: if you post everything before the first Fixing dependencies on statement, that would help too

jjeising commented 7 years ago

@chearon I did: https://p.rrbone.net/paste/fu7ovGaG#6YAbV4Jz (/usr/local/lib/libatk-1.0.0.dylib does not appear in the dylibbundler output at all).

chearon commented 7 years ago

that's very weird, it doesn't look like /usr/local/lib/libatk-1.0.0.dylib was even picked up at all in your main binary... what's the otool -L of the main binary before you run the bundler?

jjeising commented 7 years ago

That's because it's not in the main binary but only in libatkmm (all posts above about links and pathes still apply):

$ otool -L [old-binary] | grep atk
    /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/opt/atk/lib/libatk-1.0.0.dylib (compatibility version 22210.0.0, current version 22210.1.0)
$ otool -L /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib | grep atk
/usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib:
    /usr/local/opt/atkmm/lib/libatkmm-1.6.1.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/local/lib/libatk-1.0.0.dylib (compatibility version 21810.0.0, current version 21810.1.0)

I'm not sure how one gets in this situation, it may be possible to clean this up by selective reinstall or complete rebuild, but I still think diylibbundler should be able to resolve these.

chearon commented 7 years ago

I swear it worked for my dependencies of dependencies... maybe because they weren't symbolic links though. I'll look into it and open another PR if I can figure it out

chearon commented 7 years ago

@jjeising I just finished making my own bundler in python 3 here based largely on this code. I'd be interested to hear if it works for your unique situation. I was not able to find anything wrong with macdylibbundler though

jjeising commented 7 years ago

@chearon I've tested macpack and it works fine. It by the way takes a fraction of the time, but that's maybe related to the output, too.

chearon commented 7 years ago

Awesome! Yeah that and probably partly because I used async/await on the processes, so it launches a bunch of them in parallel and exits pretty fast

programmingkidx commented 7 years ago

I am having a similar problem with macdylibbunder that jjeising is having. The problem is with a file called libhogweed.2.5.dylib. It comes with nettle. If you 'brew install nettle', you would see it. I copied the libhogweed.2.5.dylib file into its own folder, then tried this command on it to copy all its prerequisites into the same folder: macdylibbundler-master/dylibbundler -od -b -x /Users/john/Desktop/erase/libhogweed.2.5.dylib -d /Users/john/Desktop/erase/libs/

This is the error I saw:

Fixing dependencies on /Users/john/Desktop/erase/libhogweed.2.5.dylib install_name_tool -change /usr/local/Cellar/nettle/2.7.1/lib/libhogweed.2.5.dylib @executable_path/../libs/libhogweed.2.5.dylib /Users/john/Desktop/erase/libhogweed.2.5.dylib install_name_tool: can't open input file: /Users/john/Desktop/erase/libhogweed.2.5.dylib for writing (Permission denied) install_name_tool: can't lseek to offset: 0 in file: /Users/john/Desktop/erase/libhogweed.2.5.dylib for writing (Bad file descriptor) install_name_tool: can't write new headers in file: /Users/john/Desktop/erase/libhogweed.2.5.dylib (Bad file descriptor) install_name_tool: can't close written on input file: /Users/john/Desktop/erase/libhogweed.2.5.dylib (Bad file descriptor)

The permissions of the copied and original files were -r--r--r--. Adding writability like this 'chmod +w libhogweed.2.5.dylib' fixed the problem. I suggest after macdylibbundler has copied a dylib, it changes that file's permissions to be writable.

clobber commented 7 years ago

I have what looks like a similar issue with complex dependencies.

First I installed corsixth via homebrew brew install corsixth

Then I ran dylibbundler -od -b -x ./CorsixTH.app/Contents/MacOS/CorsixTH -d ./CorsixTH.app/Contents/libs/ with the following output here: https://gist.github.com/anonymous/95278fe61998b9c4e0718beb365e6e84

But, while it appears all the required dylibs were copied over, I launch the app and one of the dylibs is still referencing one of its own dependencies to an absolute path:

Termination Reason:    DYLD, [0x1] Library missing

Application Specific Information:
dyld: launch, loading dependent libraries

Dyld Error Message:
  Library not loaded: /usr/local/Cellar/ffmpeg/3.3.1/lib/libavcodec.57.dylib
  Referenced from: /private/var/folders/dv/x3pz4d715l5fjdq6_3_0kr1m0000gn/T/AppTranslocation/38852360-E545-4953-9D6A-5BCAFF593E5D/d/CorsixTH.app/Contents/libs/libavformat.57.71.100.dylib
  Reason: image not found

Flaise commented 6 years ago

I'm running into a problem similar to what's already been brought up in this thread. What's the status of this issue? If that commit in the link will work, it'd be great to see that merged and published.

auriamg commented 6 years ago

Sorry, I no longer have the time to actively develop dylibbundler, also I don't have a good test case to reproduce this on my old macbook. However, if someone wants to diagnose what's going on, pull requests are welcome

auriamg / macdylibbundler

dylibbundler is missing dependencies #18