Support auto linking on Apple targets

madsmtm commented 6 months ago

Problem statement

Using a statically compiled Rust library from another language / with other toolchains can be somewhat troublesome, since the Rust library may have linking requirements that the user then has to manually pass to the linker.

Specific example: I have a static Rust library compiled for macOS that uses Winit and Wgpu and exposes a rust_main function, which I then pass to Xcode to link together with a small C main file that calls into Rust. Under the hood, Winit/Wgpu requires linking to AppKit, Metal and other such system frameworks. I'm required to also specify in Xcode the frameworks that my binary requires.

Apart from being difficult to set up (as I, the user, has to know enough about linking to understand the inscrutable error messages), this workflow is also a possible SemVer hazard, as a library like Winit cannot add a dependency on a new the system library in a minor version without possibly breaking the user's build.

This can be fixed by using the --print=native-static-libs rustc flag, but that probably requires more work for the user to integrate into the workflow than the quick fix of "just link to the broken library", and is also not very discoverable.

Proposed solution

Swift and Clang have the concept of "auto linking", where the compiler will instruct the linker to link to the correct external libraries if the user imports code from one of these libraries. This is enabled in Clang with -fmodules, and can be further controlled with the -fno-autolink flag.

On the surface, this provides much the same functionality as Rust with its #[link(...)] attribute, but there's an important distinction: the dependency information is embedded in the object file, and understood by the linker itself, which allows it to work without Swift/Clang driving the linker invocation!

This would fix https://github.com/rust-lang/rust/issues/110143.

Implementation notes

Mach-O binaries uses the LC_LINKER_OPTION load command for this, which is understood by LLVM's lld, and Apple's ld64. A few resources on that:

ELF binaries linking with LLVM's lld can use the .linker-options section or the .deplibs section.

As you can see, this is unfortunately quite platform-specific, and depends on the capabilities of the linker, so it probably isn't solvable in the general case; but I'd argue that this is still something that we could slowly improve support for, since this has a clear graceful fallback.

I'd be willing to (attempt to) implement this if you think this is desired?

Drawbacks / Unknowns

More complex linking integration, would this work for "Rust lib depending on Rust static lib" use-case, if that's even possible today?

Linker arguments are inherently ordered, and may be unexpectedly jumbled by this transformation? Would have to be properly researched and tested.

Sometimes, the user may want to opt-out of this behaviour. This can be done with the linker flag -ignore_auto_link, though we should probably document that somewhere once this has been implemented.

madsmtm commented 6 months ago

CC people whom I know have worked with macOS linking before: @BlackHoleFox, @bjorn3, @simlay

bjorn3 commented 6 months ago

This would be great to have! --print=native-static-libs is a workaround for static libraries not being able to list the dependencies they have on most platforms. We should probably create a separate object file with this information which is only added to staticlibs. This way rustc still has control over all linker args when it itself invokes the linker.

For ELF I recently found the __.LIBDEP archive member which was introduced not too long ago: https://sourceware.org/binutils/docs/ld/libdep-Plugin.html This only works with quite recent ld.bfd versions, not lld or mold afaik. As such using it is not an option, just like using LLVM's .deplibs section is not an option due to only working with lld.

ChrisDenton commented 6 months ago

On Windows we could embed static dependencies in static libraries (using the .drectve section) instead of requiring them to be passed on the command line, but Rust doesn't currently.

BlackHoleFox commented 6 months ago

Seems neat to me. Is auto-linking available in ld for every XCode version we support (XCode 9 min iirc)?

madsmtm commented 6 months ago

Is auto-linking available in ld for every XCode version we support (XCode 9 min iirc)?

Just checked, LC_LINKER_OPTION is parsed since revision 224.1 of ld/ld64, which is included in Xcode 5.0.

madsmtm commented 6 months ago

We should probably create a separate object file with this information which is only added to staticlibs. This way rustc still has control over all linker args when it itself invokes the linker.

Agreed.

For ELF I recently found the __.LIBDEP archive member which was introduced not too long ago: https://sourceware.org/binutils/docs/ld/libdep-Plugin.html This only works with quite recent ld.bfd versions, not lld or mold afaik. As such using it is not an option, just like using LLVM's .deplibs section is not an option due to only working with lld.

I think I'd argue that:

This will not be solvable in the general case, since not all linkers will support this.
But for the linkers that do support it, we should emit the information, since it will still help the users of those linkers, and won't interfere with the others.

In this specific case, it's a bit more difficult, as we'd have to choose one of these options (__.LIBDEP or .deplibs), since some linkers might start supporting both, and then the argument order would be jumbled up; so I'll probably file some issues with the linkers themselves, to see which one they'll agree on using.

madsmtm commented 6 months ago

On Windows we could embed static dependencies in static libraries (using the .drectve section) instead of requiring them to be passed on the command line, but Rust doesn't currently.

Hmm, I don't know much about Windows, are there different linkers there, and what are their names? Seems like LLVM's lld COFF supports the section at least.

bjorn3 commented 6 months ago

But for the linkers that do support it, we should emit the information, since it will still help the users of those linkers, and won't interfere with the others.

Rustc doesn't know which linker will be used and the build system that invokes the linker likely allows picking a linker which doesn't support this feature. As such it has to respect --print native-static-libs anyway. And if the linker does support it and we were emitting the section/archive member to pass the linker args, then having the build system respect --print native-static-libs too would duplicate the linker args. If we knew for sure the linker supported it, like we do on macOS, then --print native-static-libs can simply return an empty list, but for ELF that is not the case.

rust-lang / rust