Open marmijo opened 1 month ago
The volumes get cleaned up by https://github.com/coreos/fedora-coreos-pipeline/blob/ddadc038aa99692b346b422c21ede0436cd55de3/multi-arch-builders/builder-common.bu#L81, which runs daily. But I think what can happen is if too many jobs fail too quickly, we blow through the 200G limit before we even make it to the next prune.
We've been hitting storage issues on the aarch64 multi-arch builder lately and it's causing our builds to fail with a message similar to, but not limited to, the following:
I was able to log into the aarch64 builder today as the builder user and I found
/sysroot
at 100% usage.I freed up some space today by running
podman volume prune
after noticing that most of the storage space was being used by those volumes.Hopefully this will be mitigated once we redeploy the multi-arch builders on AWS and increase the size of the disk to at least 600GB from 200GB. While not necessary to redeploy the builder, landing https://github.com/coreos/fedora-coreos-pipeline/pull/986 would make it much easier. However, it might be worth exploring if we can reduce/prevent the number of dangling volumes on the builders.