search

fishfolk / bones

An easy-to-use game engine for making real games.
https://fishfolk.org/development/bones/introduction/
Other
236 stars 20 forks source link

CI: Large cache will cause runner to run out of space #480

Open MaxCWhitehead opened 1 month ago

MaxCWhitehead commented 1 month ago

First saw some of this and discussed on https://github.com/fishfolk/bones/pull/469 -

This job just failed due to a large cache being unzipped running out of space on actions runner. https://github.com/fishfolk/bones/actions/runs/11241667423/job/31253700224

While the job seems to just abruptly terminate, at the top there is a hidden "Annotions 1 error" drop down, which shows:

System.IO.IOException: No space left on device : '/home/runner/runners/2.320.0/_diag/Worker_20241008-184020-utc.log'

Perhaps we are caching too much or caching too widely across different CI jobs (maybe some do not overlap in build artifacts as much as we think?) - maybe should look at other large rust projects and see how they approach caching.

CC: @nelson137 @zicklag

MaxCWhitehead commented 1 month ago

We could look into: https://github.com/Swatinem/rust-cache but might take a bit of doing to figure out best configuration.

zicklag commented 1 month ago

I wonder if maybe we're caching between two jobs that use different versions of Rust. Like we're building the project with nightly, and then with stable or something and using the same cache.

nelson137 commented 4 weeks ago

I also wonder if there's any caches we can combine? I don't think we can do that with any of the target-specific ones (x86 & wasm32) but like the check dependencies job, build docs, etc. might be able to share. It looks like we wouldn't save more than 1 GB but it might be a step in the right direction.

MaxCWhitehead commented 3 weeks ago

Combining caches seems like a good idea in general for reducing total cache usage - but considering we are having issues due to a single cache growing beyond >4gb - not sure if this will help or make things worse? (In that a merged cache key might hit 4GB faster, but if it is the same stuff in it - probably good to do).

I don't know the answer though, definitely open to any experimentation here. Looking for anything like this that might grow cache unnecessarily might help alongside merging caches though:

I wonder if maybe we're caching between two jobs that use different versions of Rust. Like we're building the project with nightly, and then with stable or something and using the same cache.

MaxCWhitehead commented 2 weeks ago

This might also be something to look at using - https://github.com/Leafwing-Studios/cargo-cache

It includes the rust version in cache key - and additionally has a sweep feature that removes artifacts in cache that were not used in job run, pruning older artifacts from cache. Might help with the growth of cache over time?