Mergeable libraries tracking issue

[keith]

Note: this is beta 1 so things could change

As part of Xcode 15 the included linker now supports a new concept called "mergeable libraries". Mergeable libraries are dynamic libraries with an extra load command that contains enough info to re-link the library similar to a static library. Meaning that you can more (potentially) easily have dynamic libraries for development (to avoid monolithic re-links when you change a single library), but not pay the launch time cost of dynamic libraries in production builds. Today some folks using bazel achieve this by conditionally including their dynamic libraries in production vs debug using select(), but it's possible this new approach makes things easier.

The way these currently work is by providing a few different linker flags throughout the build process:

• when producing the dynamic binary that you want to be merged into another binary later in the build process, you pass -make_mergeable to the linker
• when consuming a dynamic binary that you want to be merged into your binary, you pass -merge-lNAME, -merge_library path/to/library, or -merge_framework NAME
• when consuming a dynamic binary that your consumers depend on, but you don't want to merge into your binary (to avoid the link time overhead), you can re-export that binary with -reexport-lNAME, -reexport_library path/to/library, or -reexport_framework NAME (bazel might not need to handle this one)

I think there are also a few different levels of potential integration that the rules could do for this, depending on how useful this new feature is vs doing this conditionally as you can do today.

1. Third party frameworks that are vendored to you as dynamic binaries might start containing the extra info needed to link them as a mergeable library. In this case they either need to be linked as a mergeable library (maybe only in release builds), or stripped since the extra info is sizable (potentially doubles the binary size). I think we have to make sure at least that they are stripped to not balloon app size. If you actually wanted to link them as a mergeable library you could do this today with linkopts = ["-Wl,-merge_framework,NAME"].
2. Add support for making current dynamic binaries mergeable, such as ios_dynamic_framework and ios_framework, and then consuming targets that are known to be mergeable from other binary targets with the correct linkopts (potentially conditionally). This would be roughly equivalent to just adding -make_mergeable on the targets, and then propagating some new info to indicate that was the case to add the -merge_framework argument further up the dependency tree.
3. Provide a new rule for using this behavior. It's possible the current design of requiring frameworks = [] to be used to consume ios_framework targets is too cumbersome here, especially if the line between static and dynamic frameworks is no longer as meaningful, in which case maybe we should have a different rule for this new case that is always used via deps.

Xcode also suggests mixing mergeable and dynamic libraries when you want to reduce app size when using extensions. This is equivalent to what you could do today by linking multiple static libraries into a single ios_framework that was shared between an app and extension, but also potentially complicates how we support these.

[keith]

It looks like for strip we have to pass -no_atom_info explicitly to strip the new mergeable libraries info

[luispadron]

Add support for making current dynamic binaries mergeable, such as ios_dynamic_framework and ios_framework, and then consuming targets that are known to be mergeable from other binary targets with the correct linkopts (potentially conditionally). This would be roughly equivalent to just adding -make_mergeable on the targets, and then propagating some new info to indicate that was the case to add the -merge_framework argument further up the dependency tree.

I think this makes the most sense with at least how I understood this to work in Xcode, i.e. toggling the mergeable library xcconfigs for your existing frameworks.

[nrbrook]

I stumbled upon this issue when trying to understand the intricacies of mergeable libraries, and wanted to add for reference:

• -no_atom_info option on strip is only available in Xcode 15 beta
• To determine if a library contains mergeable info, you can use otool -hl <dylib file> | grep LC_ATOM_INFO. Again, this only works with otool in Xcode 15 beta

The new versions of the tools can be found in /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

[sanju-naik]

I was trying this feature but it didn't work for me. I am able to produce a mergeable dynamic framework by passing -make_mergeable but merging it or reexporting in the App bundle is not working.

Merging

• My framework is getting merged into main App binary when I pass linkopts = ["-Wl,-merge_framework,NAME"] to ios_application [I can see symbols] but it still keeps Framewroks folder inside App bundle, ideally it shouldn't keep Frameworks folder as the deps are already merged.

Reexporting

This feature is not working even when I pass linkopts = ["-reexport_framework Name"] to ios_application. It still shows it in Frameworks.

How to reproduce

Clone this repo - https://github.com/sanju-naik/Bazel-Example checkout mergeable-libs branch and run this command bazel build --config=Debug --ios_multi_cpus=sim_arm64 --xcode_version=15.0.0 --verbose_failures BazelDemo

If someone can take a look at this, that would be great! Thanks in advance.

[keith]

Ah yea I guess nothing today stops the rules from copying all dynamic frameworks in the dependency tree. You could add an ipa_post_processor to fix that but that's a bit risky too

[sanju-naik]

Ah yea I guess nothing today stops the rules from copying all dynamic frameworks in the dependency tree. You could add an ipa_post_processor to fix that but that's a bit risky too

Got it. then I guess we need to add support for this in rules?

This feature is not working even when I pass linkopts = ["-reexport_framework Name"] to ios_application. It still shows it in Frameworks.

Also, any idea why the reexporting is not working? Is this also something rules need to add support?

[keith]

Also, any idea why the reexporting is not working? Is this also something rules need to add support?

It might be doing what it's supposed to even though the framework was copied. You can inspect the binary you passed that flag to with otool -l to see if the re-export was included.

[sanju-naik]

Yeah when I inspected the binary using otool -l /path/to/binary | grep REEXPORT, it does show cmd LC_REEXPORT_DYLIB this means its loading the reexported binary. so this part is sorted.

But since its missing ReexportedBinaries directory inside App bundle so mostly the build would crash on the device? The binary which is part of Frameworks doesn't contain any symbols [I inspected using nm -j] and looks like on the simulator it's loading binary from the .framework directory outside of the bundle so it works.

[sanju-naik]

The mergeable binary kept part of the Frameworks directory in the Bundle is of size 35 KB always [irrespective of module size] and mostly contains mergeable metadata I think, It doesn't contain any symbols.

[parvez-keeptruckin]

@keith / @sanju-naik Any idea how Google's MLKit/MLKitBarcodeScanning will be treated here given that these are added as aggregate target by cocoapods? Are these supposed to be prefixed with -merge_framework or continue with -framework? Using the -framework crashing the app when trying to scan barcode in release and debug builds both.

[keith]

A quick look at the latest MLKitBarcodeScanning shows it doesn't have mergable libraries support at this point, are you sure the crash is related?

[parvez-keeptruckin]

Actually if I do not enable MERGEABLE_LIBRARY for any pod framework [as there are other pods also besides MLKit] during post install hook then the barcode scanning works properly. Sorry but I haven't included any of MLKit libraries as MERGEABLE_LIBRARY. below is the exception I am getting *** Terminating app due to uncaught exception 'NSInvalidArgumentException', reason: '+[MLKITx_GMVUtility grayPixelDataFromBGRA:]: unrecognized selector sent to class 0x107056c10'

[keith]

That does not appear to be related. It looks like you're missing some portion of that library (maybe missing -ObjC somewhere). As far as we currently know the worse case scenario from us not supporting this at the moment is your final app binary could be larger than expected

[parvez-keeptruckin]

@keith When we do pod install after adding pod 'GoogleMLKit/BarcodeScanning', '3.2.0' then it also installs below pods:

Installing GTMSessionFetcher (1.7.2) Installing GoogleDataTransport (9.3.0) Installing GoogleMLKit (3.2.0) Installing GoogleToolboxForMac (2.3.2) Installing GoogleUtilities (7.12.0) Installing GoogleUtilitiesComponents (1.1.0) Installing MLImage (1.0.0-beta3) Installing MLKitBarcodeScanning (2.2.0) Installing MLKitCommon (8.0.0) Installing MLKitVision (4.2.0) Installing PromisesObjC (2.3.1) Installing Protobuf (3.25.2) Installing nanopb (2.30909.1)

Some of them are frameworks while others are not. Shouldn't we make those frameworks as MERGEABLE_LIBRARY?

[keith]

Either way that shouldn't be related to bazel?

[parvez-keeptruckin]

Thanks @keith for pointing out about -ObjC, I initially assumed that this field will come as part of $(inherited).

[asavill]

Yeah when I inspected the binary using otool -l /path/to/binary | grep REEXPORT, it does show cmd LC_REEXPORT_DYLIB this means its loading the reexported binary. so this part is sorted.

But since its missing ReexportedBinaries directory inside App bundle so mostly the build would crash on the device? The binary which is part of Frameworks doesn't contain any symbols [I inspected using nm -j] and looks like on the simulator it's loading binary from the .framework directory outside of the bundle so it works.

@sanju-naik Did you ever figure out why the ReexportedBinaries weren't showing up? I've tested mergeable libraries in a sample project and it works, but as soon as I try to integrate it into an existing project it fails for a typical linker error because it can't find the reexported binaries. It's not creating them OR looking for frameworks in that location.