search

mitre / hipcheck

Automatically assess and score software repositories for supply chain risk.
https://mitre.github.io/hipcheck/
Apache License 2.0
62 stars 3 forks source link

chore: Make a dummy change to main() to trigger a compile, to test rust cache #204

Closed cstepanian closed 1 month ago

cstepanian commented 1 month ago

Please don't actually merge this change.

I'll add commits to this to check whether and to what extent rust-cache successfully caches the built dependencies.

Without changing Cargo.toml or Cargo.lock, there should be an "exact match" for the cache key. If I do change one of those, I want to see whether rust-cache preserves anything about the cache.

In any case, caches created for this Pull Request should not end up being usable on the main branch due to GitHub Actions isolation rules for caches. https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#restrictions-for-accessing-a-cache

cstepanian commented 1 month ago

Test 1 Results: CI Test run: https://github.com/mitre/hipcheck/actions/runs/9896392105?pr=204

As expected, no cache is found as this is the very first use of the cache. rust-cache saved a cache for each OS. They are 330 MB for Mac and Windows each, and 400 MB for Ubuntu. GitHub sets a limit of 10GB for all caches in total, and will evict old caches to keep under the limit.

Look here for caches: https://github.com/mitre/hipcheck/actions/caches

Sample rust-cache logs on ubuntu run:

Run swatinem/rust-cache@v2
  with:
    key: ubuntu-latest
    prefix-key: v0-rust
    cache-targets: true
    cache-all-crates: false
    save-if: true
    cache-provider: github
  env:
    RUSTFLAGS: -Dwarnings
    CARGO_TERM_COLOR: always
    CARGO_HOME: /home/runner/.cargo
    CARGO_INCREMENTAL: 0
Cache Configuration
  Cache Provider:
      github
  Workspaces:
      /home/runner/work/hipcheck/hipcheck
  Cache Paths:
      /home/runner/.cargo
      /home/runner/work/hipcheck/hipcheck/target
  Restore Key:
      v0-rust-ubuntu-latest-test-a77bf122
  Cache Key:
      v0-rust-ubuntu-latest-test-a77bf122-5de08b75
  .. Prefix:
    - v0-rust-ubuntu-latest-test
  .. Environment considered:
    - Rust Version: 1.79.0 x86_64-unknown-linux-gnu (129f3b9964af4d4a709d1383930ade12dfe7c081)
    - CARGO_HOME
    - CARGO_INCREMENTAL
    - CARGO_TERM_COLOR
    - RUSTFLAGS
  .. Lockfiles considered:
    - /home/runner/work/hipcheck/hipcheck/.cargo/config.toml
    - /home/runner/work/hipcheck/hipcheck/Cargo.lock
    - /home/runner/work/hipcheck/hipcheck/hipcheck-macros/Cargo.toml
    - /home/runner/work/hipcheck/hipcheck/hipcheck/Cargo.toml
    - /home/runner/work/hipcheck/hipcheck/xtask/Cargo.toml

... Restoring cache ...
No cache found.

Post run:

... Cleaning /home/runner/work/hipcheck/hipcheck/target ...
... Cleaning cargo registry (cache-all-crates: false) ...
... Cleaning cargo/bin ...
... Cleaning cargo git cache ...
... Saving cache ...
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/hipcheck/hipcheck --files-from manifest.txt --use-compress-program zstdmt
Cache Size: ~402 MB (421330469 B)
Cache saved successfully
cstepanian commented 1 month ago

Test 2 is a build where the only change is to the source code in Hipcheck's main.rs. Test 2 Results: CI Test run: https://github.com/mitre/hipcheck/actions/runs/9897170710?pr=204

Speedup stats:

It's exciting to see build times under a minute (except for Windows, which somehow took almost 2 minutes).

As expected, since the code change did not change the Cargo.toml or Cargo.lock, rust-cache created the exact same cache key for the workspace, and therefore got an exact match on the cache when restoring. Since it was an exact match, rust-cache did not write any caches after the runs. But GitHub did track that the caches were used, which resets the expiration timer on them (GitHub will delete caches that are unused for 7 days, and this is not configurable).

Sample rust-cache logs on ubuntu run:

... Restoring cache ...
Received 0 of 421330469 (0.0%), 0.0 MBs/sec
Received 79691776 of 421330469 (18.9%), 38.0 MBs/sec
Received 176160768 of 421330469 (41.8%), 55.9 MBs/sec
Received 281018368 of 421330469 (66.7%), 66.9 MBs/sec
Received 381681664 of 421330469 (90.6%), 72.7 MBs/sec
Cache Size: ~402 MB (421330469 B)
/usr/bin/tar -xf /home/runner/work/_temp/8fde0b25-3928-457e-9bdf-dac586f730ba/cache.tzst -P -C /home/runner/work/hipcheck/hipcheck --use-compress-program unzstd
Received 421330469 of 421330469 (100.0%), 66.9 MBs/sec
Cache restored successfully
Restored from cache key "v0-rust-ubuntu-latest-test-a77bf122-5de08b75" full match: true.

Post run:

Post job cleanup.
Cache up-to-date.
cstepanian commented 1 month ago

Test 3 is a build where the only change was to the description in Cargo.toml. Test 3 Results: CI Test run: https://github.com/mitre/hipcheck/actions/runs/9897734484?pr=204

Slowdown compared to exact match:

But these times are still way better than without the cache.

(These comments are based on rust-cache 2.7.3, released on Jan 14 2024)

Based on reading source code in rust-cache, I was under the impression that the entire target directory would be wiped in the case of a cache mismatch. But fortunately that's not true -- it actually just removes packages that are no longer dependencies. This confusion was due to code deleting all regular files in the target directory, but I hadn't seen that this does not include the profile directories under it. There is separate code for cleaning profile directories (like debug or release), which keeps the contents of build and deps that are still used.

Additionally, it is only when restoring from a non-exact match that rust-cache will delete files older than 1 week, based on mtime. It seems that this could result in needlessly deleting build artifacts that are still usable. Not all dependency crates should need to release new versions every week. The cleanup functionality isn't configurable at all in this version of rust-cache.

On the positive side, this seems like it could prevent the cache from growing endlessly in the case of an error in rust-cache's assumptions about which build artifacts are used.

Sample rust-cache logs in ubuntu run: (Note that the Cache Key near the beginning doesn't exactly match the "Restored from cache key" line)

  Cache Provider:
      github
  Workspaces:
      /home/runner/work/hipcheck/hipcheck
  Cache Paths:
      /home/runner/.cargo
      /home/runner/work/hipcheck/hipcheck/target
  Restore Key:
      v0-rust-ubuntu-latest-test-a77bf122
  Cache Key:
      v0-rust-ubuntu-latest-test-a77bf122-bb2936ea
  .. Prefix:
    - v0-rust-ubuntu-latest-test
  .. Environment considered:
    - Rust Version: 1.79.0 x86_64-unknown-linux-gnu (129f3b9964af4d4a709d1383930ade12dfe7c081)
    - CARGO_HOME
    - CARGO_INCREMENTAL
    - CARGO_TERM_COLOR
    - RUSTFLAGS
  .. Lockfiles considered:
    - /home/runner/work/hipcheck/hipcheck/.cargo/config.toml
    - /home/runner/work/hipcheck/hipcheck/Cargo.lock
    - /home/runner/work/hipcheck/hipcheck/hipcheck-macros/Cargo.toml
    - /home/runner/work/hipcheck/hipcheck/hipcheck/Cargo.toml
    - /home/runner/work/hipcheck/hipcheck/xtask/Cargo.toml

... Restoring cache ...
Received 234881024 of 421330469 (55.7%), 224.0 MBs/sec
Cache Size: ~402 MB (421330469 B)
/usr/bin/tar -xf /home/runner/work/_temp/4aa1d955-49a9-4723-91b1-e52937a9f19c/cache.tzst -P -C /home/runner/work/hipcheck/hipcheck --use-compress-program unzstd
Received 421330469 of 421330469 (100.0%), 200.8 MBs/sec
Cache restored successfully
Restored from cache key "v0-rust-ubuntu-latest-test-a77bf122-5de08b75" full match: false.

Not shown in the logs: after the log "Restored from cache key ...", rust-cache silently runs cleanTargetDir, since it was not an exact key match.

Post run:

... Cleaning /home/runner/work/hipcheck/hipcheck/target ...
... Cleaning cargo registry (cache-all-crates: false) ...
... Cleaning cargo/bin ...
... Cleaning cargo git cache ...
... Saving cache ...
/usr/bin/tar --posix -cf cache.tzst --exclude cache.tzst -P -C /home/runner/work/hipcheck/hipcheck --files-from manifest.txt --use-compress-program zstdmt
Cache Size: ~402 MB (421285023 B)
Cache saved successfully
alilleybrinker commented 1 month ago

@cstepanian this is great! Very happy to see these reductions in CI time. Do you think there are more experiments to run, or should we close this with its job now done?

alilleybrinker commented 1 month ago

Closing since the experimentation is done.