Use lld by default on x64 Ubuntu 20.04 LTS

[Gankra]

This is a metabug, constraining the unbound scope of #39915.

What is lld

A linker that's part of the llvm project, which is desirable for two reasons:

• it's very friendly to cross compilation (hence its emphasis for embedded targets)
• it's very fast (often runs in half the time as Gold -- linking can take several minutes for big projects (rustc, servo, etc.) and linking can be a huge % of the compile with incremental builds, so halving this runtime is a Big Deal.)

Rust currently ships its own copy of lld which it calls rust-lld. This is used by default to link bare-metal targets and wasm.

Goal

The goal of this metabug is to use rust-lld by default on a major x64 linux distro. I have arbitrarily chosen Ubuntu 20.04 LTS as our target. With all likelihood, this will incidentally get it working/enabled on many other linux distros, BSDs, and other similar platforms, but I am purposefully constraining the scope to one distro for the sake of focusing the effort.

My understanding is that the ELF backend is quite well maintained. Quoting lld's own landing page:

We are currently working closely with the FreeBSD project to make LLD default system linker in future versions of the operating system, so we are serious about addressing compatibility issues. As of February 2017, LLD is able to link the entire FreeBSD/amd64 base system including the kernel. With a few work-in-progress patches it can link approximately 95% of the ports collection on AMD64. For the details, see FreeBSD quarterly status report.

Hopefully this has continued to improve, and using lld will be a slam dunk. Although I specify that we should use rust-lld, it may be necessary/desirable to detect and use system copies of lld.

Blocking Issues

• [x] #71519 - support making gcc use rust-lld
• [x] #71233 - coverage information not generated when using lld

[cuviper]

As a distro maintainer (Fedora and RHEL), I would definitely want this to use the system lld, at least when we're talking about the system rustc that I compile.

[Hello71]

why ubuntu 12.04? it is EOL for 3 years already.

[Gankra]

Oh whoops, I meant to write 20.04, the latest one.

[est31]

There is no such thing as a Ubuntu 20.04 LTS target platform (unlike wasm or the bare embedded targets you mention), but restricting it to x86_64-unknown-linux-gnu would be a good idea.

[Gankra]

yes, doing a good job with Ubuntu will likely get a lot of x86_64-unknown-linux-gnu working. but i'd rather get one specific (major) distro working first, since we're talking about integrating with system toolchains here.

[est31]

@Gankra I assumed you meant a target based check because the two other cases you mention (wasm and bare metal) are target based. Now I'm even more confused. What do you want to do? A host based check? As in: cargo or rustc trying to invoke lsb_release -a or something at its startup and enable LLD if it's Ubuntu 20.04? What if you cross compile to Windows or Mac? Or do you just want to talk to Ubuntu's rustc packagers so that they make it use LLD in the Ubuntu package?

[Gankra]

I don't intend to explicitly check for the distro/release, but rather am willing to have toolchain checks which succeed on vanilla Ubuntu 20.04 but e.g. don't succeed on the latest Arch or older Ubuntus.

So we don't need to worry about supporting using rust-lld with older versions of gcc/clang or whatever other tools, as long as our checks are solid enough to detect those things and fall back to using the native ld.

The starting point of this would be to not have this detection set up. We would instead blindly try to use rust-lld on x64 linux gnu targets if it's explicitly requested with the -C linker or -C linker-flavor options. This means using those options could lead to catastrophic failures in many configurations, but we will do our best to ensure they work on typical Ubuntu 20.04 installs (e.g. no older/weird toolchains installed).

In all likelihood this will incidentally get most major modern distro releases working well, but I would regard that as a Happy Accident, and not a priority. It would however provide a tool for distro maintainers to check if rust-lld mode works for them, which is good and useful.

After that we would add the detection or expand support until we could have the compiler try to automatically enable rust-lld without an explicit request on x64 linux gnu targets (possibly this step would be done by Cargo and not rustc). Once this is in good enough shape (everything working, not necessarily using rust-lld), we would enable this behaviour in production.

From there we will have a solid foundation for folks to push forward rust-lld support on their platforms by either making our toolchain detection more sophisticated, or by updating their distros to meet our requirements.

[est31]

@Gankra thanks for the explainer. Have there been any "please test LLD" threads? Those were done e.g. for pipelined compilation and might be helpful in this instance as well.

Also maybe one should think about adding an LLD label for issues so that they can be tracked.

[Gankra]

we should have an mvp of #71519 before we ask folks to try out linux lld support

[Gankra]

(CC to @rui314 in case they're interested in the fact that we're investigating this -- see also #71520)

[sledgehammervampire]

@Gankra I've made an MVP of #71519. What's next?

Edit: It should work for pure Rust projects. It doesn't always work when C libraries are dynamically linked.

[Logarithmus]

@1000teslas provided that C libraries have been fixed in https://github.com/rust-lang/rust/pull/86740, what's the next thing to do to finally use lld by default? It's more than a year since this issue was created!

[sledgehammervampire]

@Logarithmus I have no idea to be honest. I was pretty much just following what @Gankra suggested.

[Gankra]

@1000teslas oh wow, great work!

All that's left to do is advertise the new flag and provide some instructions for people to test out if it works correctly/and how it performs. I know there was some discussion about the final flag name, could you post instructions on how to use it here? Then someone more familiar with community rallying can encourage people to check it out.

[sledgehammervampire]

@Gankra It is used like this: -Z gcc-ld=lld as part of RUSTFLAGS.

Where should I post the instructions?

[Gankra]

I've prodded the compiler folks to do a writeup, we should be able to take care of the rest for you. I'll post a link the the internals/users thread when it's ready (hopefully very soon, sorry for the slow response on your initial ping).

[sledgehammervampire]

Unfortunately, it seems like my PR only works for a stage1 toolchain. I tried to dogfood my PR using a nightly toolchain from rustup on one of my pure-Rust projects and I get some weird errors. When I use ldd on the rust-lld from the nightly toolchain provided by rustup, I see: libLLVM-12-rust-1.55.0-nightly.so => not found, whereas I see no such dependency in my stage1 toolchain.

Then again, I'm using a "weird" distro (NixOS), so maybe it'll work on a more conventional Linux distro.

[bjorn3]

$ ls $(rustc --print sysroot)/lib
libLLVM-12-rust-1.55.0-nightly.so  librustc_driver-8922750e6d18b0c7.so  libstd-37fdd654777c296f.so  libtest-08487f8e282f19e0.so  rustlib

libLLVM-12-rust-1.55.0-nightly.so should be available in $(rustc --print sysroot)/lib. You may need to set LD_LIBRARY_PATH. Alternatively you can run rustup run nightly ldd /path/to/rust-lld.

[sledgehammervampire]

@bjorn3 Are you saying that even if my system's ldd can't find the location of libLLVM-12-rust-1.55.0-nightly.so, when rust-lld is invoked by rustc, it knows where to find libLLVM-12-rust-1.55.0-nightly.so?

Anyway, I get errors that look like

error: linking with `cc` failed: exit status: 1
<bunch of cc gobbledygook>
  = note: collect2: fatal error: execvp: No such file or directory

[bstrie]

@1000teslas I'd like to help get the word out about testing this more broadly, would you be able to join us in the compiler team's Zulip so that we can hash out the details? https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Need.20users.2Finternals.20thread.20for.20gcc-lld/near/245177775

[aminya]

I tried rustflags = ["-C", "linker=lld"], but I got a lot of errors like:

  = note: lld: error: unable to find library -lgcc_s
          lld: error: unable to find library -lutil
          lld: error: unable to find library -lrt
          lld: error: unable to find library -lpthread
          lld: error: unable to find library -lm
          lld: error: unable to find library -ldl
          lld: error: unable to find library -lc

Is this already reported or known?

[lnicola]

@aminya -linker=clang

EDIT: or better, link-arg=-fuse-ld=lld, see below.

[aminya]

@lnicola Thanks. This combination is now working for me.

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "linker=clang"]

[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "linker=lld"]

Note: linker=clang didn't work on Windows, and linker=lld didn't work on Linux.

Windows linker=clang error details

``` error: The specified procedure could not be found. (os error 127) --> C:\Users\aminy\.cargo\git\checkouts\argh-4cd5f7352e552d74\cd89f02\argh\src\lib.rs:176:9 | 176 | pub use argh_derive::FromArgs; | ^^^^^^^^^^^ error: could not compile `argh` due to previous error Caused by: process didn't exit successfully: `rustc --crate-name argh --edition=2018 C:\Users\aminy\.cargo\git\checkouts\argh-4cd5f7352e552d74\cd89f02\argh\src\lib.rs --error-format=json --json=diagnostic-rendered-ansi --crate-type lib --emit=dep-info,metadata,link -C embed-bitcode=no -C debuginfo=2 -C metadata=8f8c707a491e159a -C extra-filename=-8f8c707a491e159a --out-dir C:\Users\aminy\project\target\debug\deps -L dependency=C:\Users\aminy\project\target\debug\deps --extern argh_derive=C:\Users\aminy\project\target\debug\deps\argh_derive-6fcd7eb30e8474fa.dll --extern argh_shared=C:\Users\aminy\project\target\debug\deps\libargh_shared-03fa3a3743ff76a9.rmeta --cap-lints allow -C linker=clang` (exit code: 1) warning: build failed, waiting for other jobs to finish... ```

[mati865]

This will still use default system linker (probably ld.bfd) but called through Clang instead of GCC. To use LLD on Linux and Windows MinGW (your MSVC invocation seems fine):

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

[target.x86_64-pc-windows-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

[aminya]

This will still use default system linker (probably ld.bfd) but called through Clang instead of GCC. To use LLD on Linux and Windows MinGW (your MSVC invocation seems fine):

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

[target.x86_64-pc-windows-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

It worked on my local Linux machine, but it fails for CC dependencies inside GitHub Actions:

  = note: collect2: fatal error: cannot find 'ld'
          compilation terminated.

error: could not compile `proc-macro2` due to previous error
warning: build failed, waiting for other jobs to finish...
error: linking with `cc` failed: exit status: 1
  |

[aminya]

Adding "-C", "linker=clang" fixes the Github Action issue.

[target.x86_64-pc-windows-msvc]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

[target.x86_64-pc-windows-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=lld"]

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "linker=clang", "-C", "link-arg=-fuse-ld=lld"]

[mati865]

Right this flag only works with GCC 9 or newer and any reasonably modern Clang version.

[lnicola]

@mati865 wouldn't -C linker-flavor=ld.lld be better, at least it you have the rustup component? I couldn't make it work though.

[aminya]

It would be also good to clearly specify the requirements since neither Cargo nor Rustup ships a C compiler.

[bjorn3]

wouldn't -C linker-flavor=ld.lld be better,

That would directly invoke lld without the gcc/clang wrapper that is necessary to pass a lot of platform specific flags lile where system libraries can be found.

at least it you have the rustup component?

On nightly you can use rust's lld build using -Zgcc-ld=lld.

[Aaron1011]

What's the status of this issue?

[sledgehammervampire]

@Aaron1011 mostly working fine for me on x64 NixOS, waiting for bstrie to write a call to action https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/Need.20users.2Finternals.20thread.20for.20gcc-lld . Though I don't actually use the option that much these days, instead using -Clink-arg=-fuse-ld=lld.

[pnkfelix]

Visited during T-compiler backlog bonanza.

The work that @lqd is doing for rust-lang/compiler-team#510 are meant to be the final steps to unblocking turning lld on by default for Linux.

@rustbot label: S-tracking-impl-incomplete

[bstrie]

Now that https://github.com/rust-lang/compiler-team/issues/510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

[Nemo157]

I just tried this on https://github.com/rust-lang/docs.rs, to get the link time I used cargo rustc --bin cratesfyi -- -Zgcc-ld=lld -Ztime-passes -Ztime-passes-format=json |& rg '^time:' | cut -d' ' -f 2 | jq 'select(.pass == "link") | .time' and alternated running this with and without -Zgcc-ld=lld[^1] 10 times each, after a few warmup runs, giving timings of:

cargo rustc --bin cratesfyi -- -Zgcc-ld=lld
  mean = 1.00, sd = 0.01
cargo rustc --bin cratesfyi
  mean = 5.46, sd = 0.04

So lld is taking <20% of the time for linking in this case. The overall build time after touch src/lib.rs with the default linker is ~11s (but with pretty big variance) so that's almost a 50% re-build time reduction.

[^1]: while the MCP is closed the PRs implementing those new flags still appears to be open

[tgnottingham]

Is it still the case that lld lacks jobserver support? Is that going to be a blocker to turning it on by default?

[Firstyear]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

We've seen about 40% reduction in build times at SUSE for projects that use LLD over the default system linker. I don't have the full metrics on hand right now however.

[and-reas-se]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

My current project is a pretty typical website backend. Pretty heavy on dependencies: Web server, database pool, templating engine, etc. The build time on my local machine is pretty much cut in half, from two minutes to one minute for a clean build, when I change the linker from the system default (the gnu one I assume) to lld.

However, in CI the difference wasn't nearly as dramatic. Possibly due to the version of llvm / lld in CI being several llvm versions behind the one on my local machine.

[kamulos]

Now that rust-lang/compiler-team#510 is finished, I believe the next step here is to do some performance investigation to see if it's worth turning this on by default. If anyone would like to chime in with their results of turning this on for their Rust projects, now's the time to share your results. :)

For me, this is mostly about development experience. When building a big project incrementally, the time to build is dominated by the linking time.

[Tergeltengis]

what about aarch64-unknown-linux-musl-clang

[bjorn3]

What about it?

[Nemo157]

[...] but I am purposefully constraining the scope to one distro for the sake of focusing the effort

This issue is purely about a single target (and even a single distro within that target), any other targets would fall under the more general issue linked in the OP or their own tracking issues.

[lqd]

status update: as of #124129, rustup-distributed rustc (not distros') on x86_64-unknown-linux-gnu uses the self-contained lld linker by default, (rust-lld, itself also distributed with rustup), on nightly only for now and starting from tomorrow's nightly.