Unexpected dependency to homebrew xz library from a scie executable

[achimnol]

> otool -L ./backendai-install-macos-aarch64
./backendai-install-macos-aarch64:
        /opt/homebrew/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.9.0)
        /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

I discovered that executing my scie-built executable on one of "fresh" macOS setup fails due to a missing dylib: /opt/homebrew/opt/xz/lib/liblzma.5.dylib.

I think it should be statically linked into the scie binary.

[jsirois]

@achimnol this is a bit light on info. What scie-jump version are you using?

[achimnol]

https://github.com/lablup/backend.ai/blob/main/tools/pants-plugins/scie/subsystems.py#L18 It's 0.3.0.

[jsirois]

Since scie-jump is up to 0.13.2, could you please try that to see if the issue persists or is since fixed? 0.3.0 is very old.

[jsirois]

Wait, 0.3.0 is not a scie-jump version at all. That's a science version. Can you share your lift manifest?

[achimnol]

Ah, I'm confused where to put this issue. https://github.com/a-scie/lift/releases/ Yes, it's a "lift" version. It is auto-generated by the plugin: https://github.com/lablup/backend.ai/blob/main/tools/pants-plugins/scie/rules.py#L176-L185

[jsirois]

I can move the issue once I root cause, if you can just ensure I have all the info I need to repro later today, that would be great.

[achimnol]

Some refs:

• lablup/backend.ai#1694
• lablup/backend.ai#1680

[sureshjoshi]

@achimnol Is this a vendored version of my scie plugin?

If so, I see you added the fat fields. Are those in use on this issue?

[sureshjoshi]

Also, @achimnol - is the local liblzma getting dragged along by one of the zip dependencies? zip -> xz2 -> lzma ?

I don't know if all of those libs re-implement zip functionality, or if they bind to the system libs.

e.g. https://github.com/alexcrichton/xz2-rs/blob/main/lzma-sys/build.rs

[achimnol]

@sureshjoshi Yeah, it's a vendored version of your scie plugin, with a modification to use the latest science binary (0.3.0) and add an option to set fat or lazy version to build.

With the same build setup, the linux binary is completely static.

> ldd dist/backendai-install-linux-aarch64
        not a dynamic executable

[sureshjoshi]

At a glance: https://github.com/alexcrichton/xz2-rs/issues/101 ?

[jsirois]

There's a "static" feature I can enable. Thanks for the report @achimnol and legwork @sureshjoshi.

[jsirois]

It looks like ptex has no similar issue (neither ptex nor jump will be fully static, but the linked libs are Apple system libs in both cases and extremely stable):

$ otool -L dist/ptex-macos-aarch64
dist/ptex-macos-aarch64:
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 61040.1.3)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 2048.1.255)
    /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices (compatibility version 1.0.0, current version 1226.0.0)
    /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration (compatibility version 1.0.0, current version 1296.0.1)
    /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)

[jsirois]

Alright @achimnol sorry for the delay. There was unrelated tussle with an ongoing bug in rustfmt nightly.

The fix is released in scie-jump 0.13.3 and your science lift manifest (generated by the linked Pants plugin (I really, really do not like layers of obscurity)) should pick it up ~automatically. When the lift manifest does not specify a scie-jump version it defaults to latest. That latest value is cached, and the TTL is 5 days; so you'll need to blow away ~/.cache/science and then building again should solve since I think Pants does not cache package outputs generally ... but who knows. You may need to contend with Pants caching. Best would probably be to actually specify the scie-jump (and ptex for thin scies) version, but I'm not sure if the plugin allows lift manifest customization.

[achimnol]

Confirmed that the issue is resolved after rebuilding. Thanks!

> otool -L dist/backendai-install-macos-aarch64
dist/backendai-install-macos-aarch64:
        /usr/lib/libiconv.2.dylib (compatibility version 7.0.0, current version 7.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1336.0.0)