Linking with LLD

[bstrie]

LLVM 4.0 is shipping with LLD enabled, though AFAIK it is not yet production-ready on all platforms. I believe we've got an LLVM upgrade planned soon to resolve AVR/emscripten problems anyway, so now's the time to start determining what we might need to do to support it, how it impacts compiler performance/binary size/runtime performance compared to our usual linkers, and what platforms on which we might want to enable it by default.

Current status (2020-04-24) summarized at https://github.com/rust-lang/rust/issues/39915#issuecomment-618726211

Possible problems:

• LLD has internal parallelism, and is not documented to have jobserver support, which can lead to poor contention resolution in practice

[bstrie]

See also a PoC in #36120.

[retep998]

LLD may be a very good candidate for MinGW targets, because we currently bundle a linker with them anyway and MinGW's linker has a variety of issues ranging from a lack of ASLR to no bigobj support. If we can somehow also bring along the necessary mingw libraries when cross compiling and not just native targeting (which rustup's mingw package is currently limited to), then that would enable rust cross compilation from linux out of the box, which would be a huge improvement over the existing situation where people get MinGW from their distro and then run into issues because distros almost always use an incompatible MinGW.

LLD is not a good candidate for natively targeting MSVC targets, due to several reasons, the primary reason being the lack of debuginfo support. Cross compiling to MSVC targets requires libraries that can't be redistributed so we can't support that out of the box anyway.

[bstrie]

The tracking issue for upgrading to LLVM 4.0 is https://github.com/rust-lang/rust/issues/37609 .

[binarycrusader]

For the record lld is definitely not ready for Solaris targets. But on Solaris, there's no reason that I'm aware of to use lld instead of the native ld instead. We've already been looking at what it would take to have rust use Solaris ld on Solaris instead of using gcc for linking.

[whitequark]

@binarycrusader One reason to use lld is when building for Solaris, not on Solaris.

[japaric]

PR rust-lang/rust#40018 adds a -Z linker-flavor flag to rustc to make it possible to use LLD as a linker. That PR doesn't embed LLD in rustc but allows out of tree experimentation with it.

@binarycrusader ^ that may help with your experiment of directly using Solaris' ld instead of gcc.

[bstrie]

We now appear to be running on LLVM 4.0. @japaric , does this mean that the linker-flavor flag can now be easily used to compare and contrast LLD with the system linker?

[japaric]

@bstrie #40018 landed a few weeks ago. Since that landed one has been able to use -Z linker-flavor=ld -C linker=ld.lld to use an external LLD binary as a linker. Note that, unlike gcc, LLD doesn't know where the system libraries are so you'll have to pass the library search path to the linker using -C link-args='-L ...' if you are linking to any system library.

What LLVM 4.0 helps with is merging LLD into rustc. With that change we wouldn't require an external linker in some scenarios like linking MUSL binaries or bare metal programs. I say some scenarios because most targets require linking to system libraries where you will run into the library search path problem I mentioned above. For those targets, LLD won't work out of the box. It's not clear how and where to solve that problem and without a solution for that we can't switch to LLD for the most important (tier 1) targets, which the reduces the appeal of embedding LLD into rustc in the first place.

[whitequark]

@japaric What are the arguments against embedding (sysroot-relative library) search paths, as well as things like -lc -lpthread crt0.o, directly into rustc? After all, some component of the toolchain has to embed them as we don't have any standard for the platforms to follow, and binutils is not a good golden source of this knowledge.

The only drawback I can think of is the situation where the same triple would have different search paths on different flavors of systems (which will likely be exclusive to Linux/glibc triples, and especially bad on platforms with multilib). In that case, I believe clang snoops the OS name and hardcodes OS-specific conventions, which seems bad but probably unavoidable if one wants to distribute a single binary that runs on any Linux (and doesn't need the system linker).

[iainnicol]

@retep998 I had a brief look at lld a few months ago. I wasn't able to cross compile (cross link?) an .exe from Linux. It seemed like lld only supports platform native formats.

Hopefully I'm mistaken.

[bstrie]

Tagging this as a performance bug, since according to LLD's benchmarks it seems to outperform GNU ld by a factor of ten, and linking performance is a large component of compiler speed currently.

[bstrie]

Er, forgot to link the benchmarks: https://lld.llvm.org/#performance

(Relevant today since LLVM 5.0 has just been released.)

[tpimh]

Linking with LLD is much faster than bfd or gold, but I doubt that using it will significantly improve overall performance. Still I think this issue is important and should be a priority.

[bstrie]

@tpimh I'm actually not entirely sure whether the I-slow tag is supposed to represent runtime performance bugs or compiletime performance bugs, I was intending it as the latter. And IME when I look at time-passes output linking is usually in the top three longest phases, significantly longer than most, so even cutting linking time in half would probably be a huge win (especially for big things like Servo and rustc).

[jonas-schievink]

@bstrie I-slow is for bad runtime performance, I-compiletime is for compiler perf last time I checked

[iainnicol]

Good news for anybody interested in the obscure topic of cross linking from Linux to Windows. I said earlier it wasn't possible with lld, but that's only true for lld's ld flavor. It's possible for lld's link.exe flavour (lld-link).

Specifically for Rust we can do this today with a couple code changes.

1. We need to compile a very small subset of mingw-w64's CRT into .o object files. Namely some thread local storage init. We also need chkstk.

2. lld does not like MinGW's usual import libraries. Instead we need to build the .def files into .lib files ourselves, using lld-link or llvm-dlltool

3. Modify lld to treat IMPORT_NAME_NOPREFIX like IMPORT_NAME_UNDECORATE, because even with step 2 the .libs aren't perfect

4. Modify Rust's seh.rs to replace TYPE_INFO_VTABLE with ptr::null(). Required because the symbol ??_7type_info@@6B@ isn't defined in MinGW. Then build and install Rust.

5. Use .cargo/config to specify a custom wrapper script as the linker.

6. Our wrapper linker script should invoke lld-link mostly using the parameters it is passed. However we must make a few tweaks:

   a) Fix filename casing e.g. change AdvAPI32.Lib to advapi32.lib

   b) Modify the .def file Rust generates to prefix symbols with an extra underscore

   c) Override the entry point (/entry). Required probably due to a name mangling issue.

   d) Append the mingw-crt object files you compiled in step 1

7. Build your Rust project using xargo --target=i686-pc-windows-msvc

Doing the above steps allows me to cross compile Rust code. I can even panic and catch panics using Rust's SEH-based unwinding.

[retep998]

@iainnicol You're mixing the msvc target with MinGW bits, which is why you have to do all those weird modifications. If you just copy over the libraries from an existing VC++ installation then you can use lld-link normally without all those modifications or any MinGW bits.

[whitequark]

But I don't want to use an existing VC++ installation. There isn't even any way to get one without spending something like eight hours downloading and installing junk, much less to redistribute.

[rpjohnst]

The standalone build tools are much lighter-weight, unless that's already what you're referring to, in which case perhaps we ought to put some work into improving or recreating what MinGW did so it's actually compatible with MSVC.

[whitequark]

The standalone build tools are much lighter-weight

I haven't realized Microsoft distributes those. Could you link to them? Is there any reasonable way to extract the installation archive without actually running it i.e. is it an msi or something similar?

[rpjohnst]

Here they are: http://landinghub.visualstudio.com/visual-cpp-build-tools

Both the 2015 and 2017 versions are exes, but you may be able to convince the 2017 exe to give you what you want via this: https://docs.microsoft.com/en-us/visualstudio/install/install-vs-inconsistent-quality-network

[retep998]

If we really want to do this right for Windows, we'd first of all need https://github.com/rust-lang/rust/issues/30027 to eliminate the need for the Windows SDK or MinGW's import libraries. Then all we'd have left is to replace the CRT bits with our own pure Rust versions (math/memory functions, entry point, a few other runtime bits Rust needs) and we'd have a fully self-contained Rust toolchain that can create Windows binaries! The downside of this is that you wouldn't be able to statically link C/C++ code because that relies very heavily on linking in the appropriate CRT from either MinGW or VC++. Of course the whole point of Rust is to rewrite everything in Rust, so this isn't really much of an issue.

[rpjohnst]

Good news for anybody interested in the obscure topic of cross linking from Linux to Windows. I said earlier it wasn't possible with lld, but that's only true for lld's ld flavor. It's possible for lld's link.exe flavour (lld-link).

Looks like it should now be possible with the ld flavor as well: https://reviews.llvm.org/rL312926

[rui314]

The new lld's MinGW-compatible driver is a wrapper to the lld-link linker. It internally translates Unix-ish options to Windows-ish options and then call lld-link's entry point. I'm not sure if you guys want to use it because (except that the wrapper driver incomplete and not ready for use) it doesn't seem to make things easier unless you already have Makefiles for MinGW.

I have a (probably silly) question for you guys about cross-compilation. On Windows, all dllimport'ed symbols have DLL names from which they are imported. If you don't have any MSVC library files, how do you know which files dllimport'ed symbols are imported from?

[retep998]

I have a (probably silly) question for you guys about cross-compilation. On Windows, all dllimport'ed symbols have DLL names from which they are imported. If you don't have any MSVC library files, how do you know which files dllimport'ed symbols are imported from?

If you don't have any import libraries, then you have to either create import libraries or implement https://github.com/rust-lang/rust/issues/30027 so that winapi can do all the hard work of specifying which DLL each symbol comes from along with zaniness like ordinals. Something has to specify the mapping of symbols at link time to symbols/ordinals in DLLs, whether it be import libraries or annotations in your code.

[tamird]

After pulling in https://reviews.llvm.org/rL311734 I am almost able to bootstrap rustc using lld on macOS. There appears to be an issue with dylib metadata, which I still need to investigate.

I have a branch that resurrects https://github.com/rust-lang/rust/pull/36120; since we need that (very recent) lld fix, this is blocked on https://github.com/rust-lang/rust/issues/43370.

[sanmai-NL]

@tamird: #43370 has been closed.

[alexcrichton]

LLD has been added in https://github.com/rust-lang/rust/pull/48125 and is now shipping with tier 1 platforms (mac, linux, windows). You can test it out with -Z linker-flavor for each platform, although it's unlikely to work for most platforms by default. It does work, however, by default on MSVC. For example:

$ RUSTFLAGS='-Z linker-flavor=lld-link' cargo build

reduced Cargo's own link time from 2.5s to 1.5s, a nice improvement!

[bstrie]

@alexcrichton, what are the next steps? Ideally we'd have LLD working by default on all platforms (I have no conception of how much work this will take), and then I'd like to run compiletime/runtime benchmarks to see whether it makes sense to make LLD the default on any platforms. Especially with incremental compilation, linking performance is going to be more important than ever.

[retep998]

Especially with incremental compilation, linking performance is going to be more important than ever.

It's a shame linking performance still isn't important enough for us to do things like enabling incremental linking on platforms that support it. https://github.com/rust-lang/rust/issues/37543

[alexcrichton]

@bstrie I suppose those are the next steps, getting it working on other platforms :)

As to what that entails I'm not sure, but it already works on MSVC, I think it's far from working on MinGW/Linux, and we're pretty close on OSX. As for the cross-architecture support, I'm not sure as well. I don't expect we'll "stabilize" it for anything other than the wasm platform in the near future.

[cynecx]

@alexcrichton May I ask how you would specify "stabilizing"? Why wouldn't it be possible to "stablize" the linking with lld for the other major platforms rust supports? (eg. cross-compiling an executable from linux for macOS).

Right now, it's such a pain to cross-compile, for example the work needed to cross-compile an executable for macOS (x86_64-apple-darwin) from linux requires non-trivial steps like acquiring the xcode sdk and building the whole toolchain.

[alexcrichton]

@cynecx a good question! One I haven't thought much about. I think, however, we do not want to defacto-stabilize LLD just because we added it for another platform, it'll require time and work to get it right and expose it well.

[whitequark]

for example the work needed to cross-compile an executable for macOS (x86_64-apple-darwin) from linux requires non-trivial steps like acquiring the xcode sdk and building the whole toolchain.

LLD wouldn't really help here, you still need the Xcode SDK because it has headers you can't redistribute (and depending on what you're building, you'll need other SDK tools as well).

[rkarp]

What's really nice about LLD now being built-in on nightly is that you can easily cross-compile pure Rust projects from Windows to Linux with RUSTFLAGS='-Z linker-flavor=ld.lld' cargo build --target x86_64-unknown-linux-musl. Great for writing small tools for Linux machines you can't simply install Rust on.

[briansmith]

I don't expect we'll "stabilize" it for anything other than the wasm platform in the near future.

Like @rkarp said, a very common use case is targeting x86_64-unknown-linux-musl (and eventually steed) for supporting containerized Linux workloads. This is one of those things that Go does really well and where we seem to be very close to Rust being able to do likewise. In terms of actual usage, I bet LLD for x86_64-unknown-linux-musl would actually get used much more broadly than wasm.

More generally, when it comes to cross-building, I don't think a "it must work for all hosts and/or all targets" approach makes sense. I think it makes sense to stabilize this on a target-by-target basis as targets start working.

In particular, I would love to help with the effort to get LLD for the x86_64-unknown-linux-musl target stabilized ASAP.

[rocallahan]

My project has 37 crates, and the build links about 70 binaries (lots of tests). Unscientifically (eyeballing top) at least half of the build time we are only running ld. I expect using lld would speed up our builds a lot. We're on stable Rust and I haven't managed to get lld 6.0 to work yet.

[alexcrichton]

@briansmith have you tested out LLD for musl and your use case? In theory all you'd need to do to test it is to pass -Z linker-flavor=ld.lld, and if that works seems plausible we could switch the defaults!

@rocallahan just to confirm, y'all are using the gold linker currently, right? (as afaik it's faster than the standard binutils linker). If -Z linker-flavor=ld.lld works (and is faster) then we could look to stabilize it perhaps! What platform was that on?

[rocallahan]

Unscientifically (eyeballing top) at least half of the build time we are only running ld.

That's for a debug build BTW.

y'all are using the gold linker currently, right? (as afaik it's faster than the standard binutils linker)

No, that's the Fedora system linker, the standard GNU linker.

What platform was that on?

Fedora 27, quad-core Skylake laptop with SSD. I will get some performance numbers.

[alexcrichton]

Ah ok good to know! Debug builds will probably get the most benefit in link time from split dwarf (https://github.com/rust-lang/rust/issues/34651) rather than linker improvements.

For timing information, though, @rocallahan if you get a chance mind testing with ld.gold and ld.lld?

[rocallahan]

Sure. I should also remind that issue #48762 is very low-hanging-fruit for speeding up Linux debug link times. (We're already using a hacked linker script that drops .debug_pubnames/.debug_pubtypes from the executable.)

Split DWARF might be good but it might also cause problems for users. I'll comment in that issue.

[briansmith]

@briansmith have you tested out LLD for musl and your use case? In theory all you'd need to do to test it is to pass -Z linker-flavor=ld.lld, and if that works seems plausible we could switch the defaults!

OK, I will test things. Perhaps initially there should be a middle ground between "default" and "only on nightly," some way of opting into using LLD like we can with -Z, but without using -Z so that it works in stable builds.

[matthiaskrgr]

but without using -Z so that it works in stable builds.

You can try RUSTC_BOOTSTRAP=1 RUSTFLAGS="-Z linker-flavor=foo" cargo build

[SimonSapin]

Please let’s not recommend the bootstrap flag? I worry that if enough projects rely on it things will become de-facto stable, defeating the point of the entire stability mechanism.

[matthiaskrgr]

Sorry =/

[alexcrichton]

As a data point, linking the rustc_trans crate in the Rust repo itself plummeted from a 78s link time to 1s link time on my local machine.

[rocallahan]

My performance results from making a whitespace change to a crate near the bottom of our crate hierarchy. Quad core Skylake laptop, 16GB RAM, rustc 1.24.0, LLD 7.0.0, GNU ld 2.29-13. We use a custom linker script that discards .debug_pubnames and .debug_pubtypes; LLD uses quite different code paths when a linker script is present, so that might affect things.

GNU ld:

real    2m39.138s
user    8m18.992s
sys 1m37.513s

LLD:

real    2m19.164s
user    6m4.477s
sys 0m56.858s

Gold didn't work, it barfed on our linker script. The results are fairly stable. LLD doesn't affect the end-to-end time that much but it does reduce CPU usage significantly; I guess that means our build doesn't spend much time waiting for lds to finish, but it does spend a lot of CPU time running them.

[nnethercote]

See #50584 for a real world example where switching from GNU ld to lld makes a common "minor change and rebuild" workload execute more than 2.5x faster.

[alexcrichton]

Er https://github.com/rust-lang/rust/issues/50584#issuecomment-400918647 is more appropriate here:

---

The next step for stabilizing LLD would be to get a flag, like -Z linker-flavor=lld, working for all targets (Windows + Mac + Linux). It'd do whatever it needs to do to work across the various platforms.

Once that's done we can advertise it to the community, asking for feedback. Here we can gain both timing information as well as bug reports to send to LLD. If everything goes smoothly (which is sort of doubtful with a whole brand new linker, but hey you never know!) we can turn it on by default, otherwise we can work to stabilize the selection of LLD and then add an option to Cargo.toml so projects can at least opt-in to it.

[steveklabnik]

We have switched to lld for some targets: https://rust-embedded.github.io/blog/2018-08-2x-psa-cortex-m-breakage/

I believe for wasm as well?