ci: ensure nightly and rustup are available

[jedbrown]

@wsmoses This passed, but isn't caching (yet) or testing (yet).

What to cache?

Perhaps the right choice is to cache all of build, though it's going to be bigger than the end products of compilation. I don't think rebuilding Rust stage1 (without caching) is prohibitive so long as LLVM is available. These are the sizes for just the LLVM portion. I'll test whether we can delete build/host/llvm/build without that triggering a clean build of LLVM.

$ dust build/host/llvm
 48M   ┌── include                                   │█                                                               │   2%
 45M   │ ┌── c-index-test                            │█░░░                                                            │   1%
 97M   │ ├── llvm-exegesis                           │███░                                                            │   3%
190M   ├─┴ bin                                       │████                                                            │   6%
 47M   │ ┌── libclang.so.17.0.6-rust-1.77.0-nightly  │█░░░░░░░░░░░░░░░░░                                              │   2%
 49M   │ ├── libclangSema.a                          │██░░░░░░░░░░░░░░░░                                              │   2%
 89M   │ ├── libclang-cpp.so.17-rust-1.77.0-nightly  │██░░░░░░░░░░░░░░░░                                              │   3%
141M   │ ├── libLLVM-17-rust-1.77.0-nightly.so       │███░░░░░░░░░░░░░░░                                              │   5%
865M   ├─┴ lib                                       │██████████████████                                              │  28%
 44M   │   ┌── clang-ast-dump                        │█▒▒▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   1%
 45M   │   ├── c-index-test                          │█▒▒▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   1%
 97M   │   ├── llvm-exegesis                         │███▒▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   3%
236M   │ ┌─┴ bin                                     │█████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   8%
 38M   │ │     ┌── StaticAnalyzer                    │█▓▓▓▓▓▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   1%
 48M   │ │     │   ┌── obj.clangSema.dir             │█▓▓▓▓▓▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   2%
 48M   │ │     │ ┌─┴ CMakeFiles                      │█▓▓▓▓▓▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   2%
 49M   │ │     ├─┴ Sema                              │██▓▓▓▓▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   2%
246M   │ │   ┌─┴ lib                                 │██████▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   8%
267M   │ │ ┌─┴ clang                                 │██████▒░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │   9%
302M   │ ├─┴ tools                                   │███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░                       │  10%
 44M   │ │ ┌── CodeGen                               │█▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   1%
 47M   │ │ ├── libclang.so.17.0.6-rust-1.77.0-nightly│█▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
 49M   │ │ ├── libclangSema.a                        │██▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
 50M   │ │ ├── Transforms                            │██▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
 89M   │ │ ├── libclang-cpp.so.17-rust-1.77.0-nightly│██▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   3%
141M   │ │ ├── libLLVM-17-rust-1.77.0-nightly.so     │███▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   5%
 59M   │ │ │ ┌── RISCV                               │██▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
 62M   │ │ │ ├── X86                                 │██▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
 63M   │ │ │ ├── AArch64                             │██▓▓▓▓▓▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │   2%
319M   │ │ ├─┴ Target                                │███████▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░░░░                       │  10%
1.4G   │ ├─┴ lib                                     │█████████████████████████████░░░░░░░░░░░░                       │  45%
1.9G   ├─┴ build                                     │█████████████████████████████████████████                       │  64%
3.0G ┌─┴ llvm                                        │███████████████████████████████████████████████████████████████ │ 100%

When/how to rebuild Enzyme?

We currently have an (empty) marker file build/host/enzyme/enzyme-finished-building that controls whether Enzyme is rebuilt. Shall we change it to hold the Enzyme commit hash as returned by

$ git -C src/tools/enzyme rev-parse HEAD
7e3e90f4287068a41d3fb5a99127ee2857353b04

Note that this need not be the same as the currently-committed submodule (if the tree is dirty, which should not be the case in CI)

$ git ls-tree --object-only HEAD src/tools/enzyme
7e3e90f4287068a41d3fb5a99127ee2857353b04

I don't think caching Enzyme is important -- it doesn't take long to build and it'll be changing relatively frequently.

Actually running tests

Our current cargo +enzyme test --examples does nothing because it's running in the rustc repo. Rust has x test for testing the compiler and we could have some Enzyme tests there. We've also been writing tests in EnzymeAD/rustbook and we could check out that repo to run the tests that are there. Shall we also put ADBench tests in that repository?

[ZuseZ4]

@wsmoses I can't add @jedbrown to the rust org of Enzyme, can you do that please?

[wsmoses]

@wsmoses I can't add @jedbrown to the rust org of Enzyme, can you do that please?

Done

[ZuseZ4]

I agree that for CI it's probably no issue to rebuild Enzyme, I was never bugged by that in Enzyme core. Also not caching Rust is good because it has less conflict potential if I change Rust, which happens frequently. I never changed the LLVM submodule, at most I added more C++ and Rust wrappers for LLVM functionality. The build times for Rust are also usually in the range of 1-2 minutes (on 48 cores), so it should be well doable even in CI.

[jedbrown]

:tada: We're at 22 seconds to restore the cached LLVM build (saves about 2 hours). That should automatically rebuild when src/llvm-project is updated (historically, one or two updates per month on rust-lang/rust@master). Building Enzyme and Rust stage1 from source takes 36 minutes. Perhaps acceptable, and I think we can improve it with caching, tuned depending on what changes faster.

[jedbrown]

This is using rustbook@main now, and will cache Enzyme (as soon as we get a passing pipeline after fixing the regression). I think it's ready to merge so that any fixes or subsequent development have these tests.