Cargo pre-planning: downloads

[nrc]

• caching/distributing binaries
• custom registries (is this just stabilisation?)
• offline mode (remaining work - possible global cache)
• (longer term) cloud build/sharing intermediate artifacts

cc @withoutboats, @alexcrichton

[nrc]

ping @withoutboats and @alexcrichton

Some starter questions:

• distributing binaries seems like a big win for Cargo and for Rust in general since it could help a lot with (initial) build times. It also dovetails nicely with moving some Rustup components into Cargo.
  • how much overlap do you think there would be with Rustup stuff
  • could we design in a way that was relatively agnostic about the backend (crates.io side)?
  • how much work is this going to be in Cargo?
• what is the current status of custom registries and offline mode? Is there outstanding design or implementation work?
• do you think it is feasible/a good idea to plan for any cloud build functionality in the next year? Next three years?

[ishitatsuyuki]

Regarding cloud build: I think we may want to reuse crater artifact, so integration with crater should be in the mind.

[alexcrichton]

Sorry I haven't had a lot of time to dig into this, I was planning on revisiting/posting to this after the 2018 edition is out

[nrc]

Notes from today's Cargo meeting:

• Can we collaborate more with the compiler? Cargo knowing about the incremental cache as well as just rlibs.
• Priority of dist builds? Firefox build system, compile times. Possible we could have a big impact
• We should evaluate sccache to see how much benefit there is here
• Security? Do we need vetting and review? At first we can build ourselves and just need to trust us + the source code.
• Hardware - cost, maintenance. We could provide functionality without the build source. How much could we afford to do?
• Implementation could be tricky - what files are created and can be deleted? (we have no idea! Should be documented)
• custom registries - needs documenting, there have been changes since the RFC (who knows what?), what exactly would be stabilised (@ehuss volunteers to find out <3). Some tooling missing - generating the index, example server. How much do people want cargo publish, etc. (current UI: configure in config (registry exists), then a patch-like thing in Cargo.toml).

[alexcrichton]

Alright I've got some time now! I'd like to jot down some thoughts on this...

Accelerating builds through caching

I think one of the lowest hanging fruit for making the compiler feel faster is going to be increasing the amount of caching going on in a typical build process. While single-thread performance of the compiler is still important, it's always nice to do no work at all! There's lots of different avenues we can pursue here, and here's what I'd personally think we should tackle in order of priority:

• System-wide cached builds. Basically let's add a global target directory in your home directory, so when you build multiple projects on a system (or switch checkouts or whatever) we'll try to reuse as much as possible that's already in the cache. I'm not sure to what granularity we want cache entries there to be (right now target directory is somewhat coarse in caching), and such a strategy may also require some degree of automatic gc that Cargo does (or at least has a command to do). In any case this will help developers over time that accrue builds locally, and may help accelerate future plans for things like building libstd on-demand.

• Enabling easily-cached builds on CI. One of the pain points for Rust build times is that almost all CI systems start from scratch, doing full builds. While you can cache the target directory it wasn't ever really intended for that, and it's likely that big portions of the Rust community are all building the same thing. It'd be great if we could either improve the experience of a cached target directory on CI or otherwise provide a built-in-sccache-like solution for caching artifacts. This would go a long way towards bringing down cold build times on CI.

• Enabling caching of build scripts. One of the biggest offenders that thwarts sccache today is build scripts as it can't cache the execution of the build script. We should develop a scheme such that sccache (or something) has enough knowledge to cache the output of a build script as it honestly almost never changes!

• Enable caching of procedural macros and executables. Another major offender that thwarts sccache is that procedural macros and executables (like build script binaries) can't be cached. This is because the linking phase deals with a lot of system things like the linker itself and system libraries that aren't easily tracked across machines. Perhaps the easiest way to integrate this would be to somehow cache object files via sccache rather than entire output artifacts. Either that or somehow split this up in Cargo to "make it just work" to do a cacheable compilation first and then a always-blazingly-fast compilation second.

• Enable more fine-grained caching between Cargo and the compiler. Right now Cargo treats the compiler like a big black box, and the compiler treats the build system as a pretty large black box too. I don't know of anything off the top of my head, but I feel like these two systems are ripe for closer integration with one another. For example the compiler's incremental cache may be shareable across crate versions beneficially, or may even be shareable across some crates themselves. It's worth exploring opportunities here for use cases like "I changed one file, why do I need to download an entire incremental cache or recompile the whole crate on CI?". This step would likely naturally involve some degree of interfacing with sccache as well.

• Finally, distributing precompiled binary libraries to users. I place this last because I don't think we'll get around to it in 2019. The main reason for that is that this is a really tricky thing to do, and everything mentioned above I think will basically be a prerequisite of one form or another. For example we could do this today in theory with a few tweaks to sccache to have a global Rust community crate cache, but it honestly wouldn't really help that much without solving many of the above problems first (build scripts, procedural macros, on-by-default configuration, etc). While this is a laudable goal and would be an awesome deliverable, I'm hopeful that the above points are much more bite-sized to the point where we could actually get them done.

Affecting Cargo's download behavior

Also mentioned in the OP were a few parts about Cargo not downloading items ore otherwise downloading things from different places. I think these are quite important too, and perhaps orthogonal to the above!

• Stabilizing custom registries. This seems like a no-brainer, we need to get this over the finish line and finish up any final work items.

• Stabilize "offline mode" in one way or another. Long requested, we should push this over the finish line with appropriate tooling support if necessary.