GaloisInc / crucible

Crucible is a library for symbolic simulation of imperative programs
617 stars 42 forks source link

`crux-mir` nightly Docker build failure (`no space left on device`) #1175

Closed RyanGlScott closed 5 months ago

RyanGlScott commented 5 months ago

The nightly crux-mir Docker image CI job is currently failing with:

#55 importing to docker
#55 loading layer d101c9453715 327.68kB / 29.54MB
#55 loading layer ddbe1feb8a54 393.22kB / 36.40MB
#55 loading layer 26c679dcf4b5 557.06kB / 474.75MB
#55 loading layer 26c679dcf4b5 285.21MB / 474.75MB 2.0s
#55 loading layer 26c679dcf4b5 453.44MB / 474.75MB 4.2s
#55 loading layer 7e0bd5b881d4 557.06kB / 471.54MB
#55 loading layer 7e0bd5b881d4 53.48MB / 471.54MB 2.1s
#55 loading layer 7e0bd5b881d4 172.69MB / 471.54MB 4.1s
#55 loading layer 7e0bd5b881d4 217.25MB / 471.54MB 6.2s
#55 loading layer 7e0bd5b881d4 295.24MB / 471.54MB 8.3s
#55 loading layer 7e0bd5b881d4 362.64MB / 471.54MB 10.4s
#55 loading layer 7e0bd5b881d4 471.54MB / 471.54MB 15.0s done
#55 loading layer d101c9453715 327.68kB / 29.54MB 23.9s done
#55 loading layer ddbe1feb8a54 393.22kB / 36.40MB 22.9s done
#55 loading layer 26c679dcf4b5 472.38MB / 474.75MB 21.4s done
#55 ERROR: write /usr/local/rustup/toolchains/nightly-2023-01-23-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_macros-1f5c22ed96efe43a.so: no space left on device

#54 exporting to docker image format
#54 sending tarball 32.9s done
#54 ERROR: rpc error: code = Unknown desc = write /usr/local/rustup/toolchains/nightly-2023-01-23-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_macros-1f5c22ed96efe43a.so: no space left on device
------
 > exporting to docker image format:
------
------
 > importing to docker:
------
ERROR: failed to solve: rpc error: code = Unknown desc = write /usr/local/rustup/toolchains/nightly-2023-01-23-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_macros-1f5c22ed96efe43a.so: no space left on device
Error: buildx failed with: ERROR: failed to solve: rpc error: code = Unknown desc = write /usr/local/rustup/toolchains/nightly-2023-01-23-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_macros-1f5c22ed96efe43a.so: no space left on device

Admittedly, the crux-mir Docker image is somewhat large (it's about 1GB in size when compressed), but I don't think it's unreasonably large. Moreover, this error only started occurring recently. I suspect that we were always close to using all of the disk space that a GitHub Actions CI runner offers (this page claims that it's 14 GB), and recent changes have pushed the image size just over the limit.

At first glance, I wondered if this was a caching-related issue, so I tried clearing all of the crucible repo's caches and retrying. Unfortunately, the error persists even from a clean cache.

https://github.com/actions/runner-images/issues/2840#issuecomment-790492173 provides a workaround: you can clear certain GitHub Actions directories that contain pre-installed tools to free up about 10 GB of space:

sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY" # Python installations

I don't think the crucible CI uses any of these tools (note that we use ghcup-installed versions of GHC rather than the pre-installed ones found in /opt/ghc), so this would be worth a try.