search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.55k stars 12.74k forks source link

Move more of rustc_llvm to upstream LLVM #46437

Open alexcrichton opened 6 years ago

alexcrichton commented 6 years ago

In general we try to use the LLVM C API whenever we can as it's generally nice and stable. It also has the great benefit of being maintained by LLVM so it tends to never be a pain point when upgrading LLVM! Unfortunately though LLVM's C API isn't 100% comprehensive and we often need functionality above and beyond what you can do with just C.

For this custom functionality we typically use the C++ API of LLVM and compile our own shims which then in turn have a C API. At the time of this writing all of the C++ to C shims are located in the src/rustllvm directory across three main files: ArchiveWrapper.cpp, PassWrapper.cpp, and RustWrapper.cpp. These files are all compiled via build.rs around here where basically use llvm-config to guide us in how to compile those files.

The downside of these shims that we have, however, is that they're difficult for us to maintain over time. They impose problems whenever we upgrade LLVM (we have to get them compiling again as the C++ APIs change quite regularly). Additionally it also makes consumers of Rust have a more difficult time using custom LLVM versions. For example right now our shims compile on LLVM 5 but probably not LLVM trunk. Additionally for users that like to follow LLVM trunk then keeping up with the breakage of our shims can be quite difficult!

To help solve this problem it seems the ideal solution is to try to upstream at least a big chunk of the C++ APIs that we're using. This way we can much more closely stick to LLVM's C API which is far more stable. It makes it that much easier for us to eventually upgrade LLVM and it makes users using a custom LLVM not need to worry about using an LLVM beyond the one that we're using (aka LLVM trunk).

I'll try to have a checklist here we can maintain over time which also is a good listing of what each of the APIs does!

ArchiveWrapper.cpp

In general this is functionality for reading archive *.a files in the Rust compiler. This makes reading rlibs (which are archive files) extra speedy. The functions here are:

These functions are basically just reading and writing archives, using iterators for reading and providing a list of structs for writing.

PassWrapper.cpp

This file is when we get into a bit more of a smorgasboard of random functions rather than a consistent theme, so I'll comment more of them inline below.

A general theme here I've found as I wrote these down is that it's not critical that all of these are implemented. I could imagine that it would be possible to have a mode where we as rustc still compile shims sometimes (like the ones below) but many of the shims are stubbed out to not actually use LLVM at all if we're in "non-Rust-LLVM mode" (aka custom LLVM mode). In other words, we don't necessarily need to upstream 100% of these functions.

These APIs are all related to ThinLTO are are still somewhat in flux, there may not be a great C API just yet.

RustWrapper.cpp

Sort of even a bigger smorgasboard than PassWrapper.cpp! Note that many of these functions are very old and may have actually made their way into the C API of LLVM by now, in which case that'd be awesome!

alexcrichton commented 6 years ago

For ArchiveWrapper.cpp a fun project could also be trying to use a crate on crates.io instead!

djc commented 6 years ago

What's the status with LLVM upstream these days? How compatible is Rust with upstream LLVM?

alexcrichton commented 6 years ago

We track upstream pretty closely, but all of the above bindings occasionally require updates to make sure we compile against upstream. To that end it'd still be best to upstream this!

alexcrichton commented 5 years ago

cc @petrhosek

we were talking about this at rustconf, and while dated it still has some good information I think!

inglorion commented 3 years ago

I'm interested in working on this. Besides the benefit to Rust, I also think LLVM benefits from having a more complete C API. Since this has been open for a while: Are there any parts of this that are no longer relevant and I can avoid spending time on?

arrowd commented 2 years ago

Any progress on this? Downstream rust packagers will also greatly benefit from this by the means of much shorter build times.

bjorn3 commented 2 years ago

Why would it result in much shorter build times? These shims are small compared to the rest of rustc and LLVM and work with precompiled LLVM distributions too, not just the rust fork of LLVM.

arrowd commented 2 years ago

Oh, sorry, I thought it is a blocker for using precompiled LLVM.

pnkfelix commented 2 years ago

Visiting for T-compiler backlog bonanza.

Seems like a good list (if long) of relatively simple tasks.

@rustbot label: S-tracking-impl-incomplete