Shipping clang as a rustup component

[michaelwoerister]

Sometimes it is beneficial (or even necessary) for mixed-language projects to build C/C++ with a Clang version that matches Rust's LLVM version. Cross-language LTO is one of these cases.

Since Rust's LLVM hardly ever matches a specific release and is often ahead of the current stable release, it would be great if we provided an easy way to get ahold of the right Clang binaries. It looks like we are already building Clang for Rust-enabled LLDB anyway. Could we make also make it available via rustup? Either as a standalone component, or as part of lldb or llvm-tools?

cc @rust-lang/infra @rust-lang/release

[alexcrichton]

I personally feel like this is going perhaps a step too far in our distribution strategy in that I think we want to try to become a one-stop-shop for all your toolchain needs. Shipping a C compiler seems fraught with complications like "will we start shipping header files and runtime libraries next?"

I agree that cross-language-LTO is a good use case, but I think I'd probably prefer to document and tests that any particular version of a Rust compiler is compatible with some set of clang releases (probably just one). That way users can figure out the best way to install clang on their end (which seems like it's unlikely to be from our sources?) and we'd be testing it worked.

(I'm not 100% certain how the debugger/llvm tools stories will play out in the long run, it already feels to me like we're overstepping our bounds slightly)

[Centril]

Maybe we can try this out as a "preview" and see how it develops and if there's a big need for it?

[dwightguth]

I can say that shipping clang wouldn't really be enough for me because we also have to use the llvm API to generate code on our own. So it would definitely be better for our purposes if rust stable was compatible with some specific release of llvm.

[BatmanAoD]

It would probably be helpful for rustc to provide a programmatic way to query the recommended Clang version for LTO compatibility, then. This would provide a straightforward way for any interested parties to write a script that can install/update Rust and then install or update Clang to match.

On Fri, Nov 30, 2018, 10:33 AM dwightguth notifications@github.com wrote:

I can say that shipping clang wouldn't really be enough for me because we also have to use the llvm API to generate code on our own. So it would definitely be better for our purposes if rust stable was compatible with some specific release of llvm.

— You are receiving this because you are on a team that was mentioned. Reply to this email directly, view it on GitHub https://github.com/rust-lang/rust/issues/56371#issuecomment-443278911, or mute the thread https://github.com/notifications/unsubscribe-auth/ACNOT4Gf2J4ZuSJhYh9JNB4skpx3nHJZks5u0WvjgaJpZM4Y7XJ8 .

[bstrie]

rustc has a policy of supporting at least one stable release of LLVM in order to satisfy distro packaging, correct? Maybe rather than shipping Clang, we could provide a rustup option where you receive a version of the compiler built with a stable LLVM, which one can surely then find a suitable Clang for.

[cuviper]

Or such users can just switch to a distro rustc with its external LLVM and matching clang... :innocent:

[bstrie]

Sure, but if that's the case then this entire bug is pointless. :) No such luck for Windows users, sadly!

[nikic]

rustc has a policy of supporting at least one stable release of LLVM in order to satisfy distro packaging, correct? Maybe rather than shipping Clang, we could provide a rustup option where you receive a version of the compiler built with a stable LLVM, which one can surely then find a suitable Clang for.

The main issue with that is probably the additional CI time this would take. At the very least, it requires building an additional codegen backend (not so expensive), but probably also running the full testsuite with it at least on tier 1 platforms. (On the other hand, that would probably alleviate distributor concerns that Rust is not sufficiently tested with system LLVM.)

It also seems kind of odd to go back to an older LLVM version (and possibly lose performance or features) to make use of cross-lang LTO.

[gnzlbg]

Now that cross-lang LTO is being stabilized (https://github.com/rust-lang/rust/pull/58057) we should definitely ship rust-clang with rustup. We are pretty much always using a non-release LLVM version pinned at some particular commit, which would mean that for people to use cross-lang LTO in nightly, they would need to get the LLVM sources used by the current nightly (potentially with Rust patches), find the respective clang sources, and compile LLVM and clang (maybe also compiler-builtins and friends), just to get started with the really FUN part of the process [0].

[0] FUN: setting up a C/C++ toolchain that differs from the platform toolchains when compiling C/C++ for Rust projects, such linking against other C/C++ code compiled by other different toolchains like the platforms one still works.

[bmmoore]

@bstrie Where is that policy stated? It would really be quite useful to have rustc promise to be LTO-compatible with a release version of LLVM, or at least ship such a rustc.

[dmitris]

question regarding https://doc.rust-lang.org/rustc/linker-plugin-lto.html#toolchain-compatibility - the page lists Rust 1.34 and Rust 1.35 in the compatibility table - should we add 1.36 as well since the current rustc version is 1.36.0?

[mati865]

@dmitris it's updated on nightly: https://github.com/rust-lang/rust/blob/e562b24ae325f5a31b7ba5873e3db426a14e6342/src/doc/rustc/src/linker-plugin-lto.md#toolchain-compatibility

[dmitris]

@thanks mati865 - Go has pages like https://tip.golang.org/cmd/go/, does Rust have any such "symlink" URLs that would link to the latest/nightly version of https://doc.rust-lang.org/stable/rustc/ etc? If not, do you think it would be useful?

[mati865]

does Rust have any such "symlink" URLs that would link to the latest/nightly version of doc.rust-lang.org/stable/rustc etc?

Literally replace stable with beta or nightly in that URL :smile:

[chadbrewbaker]

Is there a script to compile and throw in rustup nightly all the llvm-project binaries for the same branch? Honestly don't care if it's "supported" I'm just tired of rust and clang fighting with each other from being on different llvm-project branches.

[leo60228]

I think there's a good argument to be made for this even without LTO. A lot of crates require a C compiler, and being able to install one via rustup would go a long way towards making rustc more usable out of the box.

[messense]

https://actually.fyi/posts/zig-makes-rust-cross-compilation-just-work/

🤪😜

[dvc94ch]

What's missing here? Is it just someone implementing it? Can this be implemented without major change proposals or rfcs?

[dvc94ch]

Looking into this, I noticed that while lldb-preview shipped for a while, it was dropped at some point during the llvm 9 upgrade. So actually shipping lldb and having a commitment from the rust team to not just drop it when convenient would be nice. At least lld seems to be shipped reliably.

• https://github.com/rust-lang/llvm-project/pull/19

[riidefi]

Beyond cross-language LTO, this has additional benefits on Windows where MSVC is outperformed by Clang wrt to compilation time, runtime performance and safety features. On some sys crates I author I note that running SET CXX=clang before cargo build on Windows is almost always preferable. Additionally, some more modern C++ projects that use "TRY" macros (like Rust's ? operator or old try! macro) only support clang/gcc, which makes writing bindings a bit complicated because not only are you forcing clang for the current -sys crate, but your project may be linked against other -sys crates built with MSVC, which can bring about extra considerations.

For people working in domains with not much Rust penetration, I think shipping a rust-clang component could make the process a lot more beginner friendly.