search

lightbend-labs / dbuild

Multi-project build tool, based on sbt.
https://lightbend-labs.github.io/dbuild
Other
83 stars 14 forks source link

automatic cleanup by count, not time #170

Closed adriaanm closed 1 year ago

adriaanm commented 9 years ago

When running dbuild in CI, it's not always predictable how many times it will run per day. Automatic cleanup is crucial to stay under 40G or so. To guarantee we won't run out of disk space, we need to be able to say "keep cache from last N runs", rather than "keep all caches from all runs in last N hours", which can vary quite a bit in size.

adriaanm commented 9 years ago

/cc @SethTisue

cunei commented 9 years ago

This feature can be implemented, but it involves some code changes. Presently, dbuild has no concept of ordering for subsequent invocations: it handles each run on its own. The time-based approach works because each extraction and build directory is time-stamped; therefore, each one can be inspected for cleanup in isolation, given the timestamp and the current time. Conversely, all of the metadata concerning builds and all of the generated artifacts are just hashed and dumped in a custom-format repository, only indexed by hash. That is also the reason why dbuild is currently unable to detect the first successful run after a string of failures, generating a message like "back to normal, now", Jenkins-style.

These features could be implemented by traversing this custom-format repository (kept under ~/.dbuild/cache-*) in order to establish the build ordering, or possibly by keeping additional indexes (bar concurrent repo accesses). Implementing a traversal of this sort was actually something that has long been planned in order to be able to perform a periodic cleanup of the repository data. Right now, for each dbuild run, the metadata and the generated artifacts are just added to the cache, but never removed; albeit slowly, the cache can therefore grow indefinitely. The plan was to inspect the list of builds whose metadata is stored in the repository, perform a traversal of the linked metadata corresponding to the extraction and build stages, and of the related artifacts, and finally remove the metadata and artifacts that are only referred to by the oldest builds. The same process will also reveal the location of the oldest extraction and build directories, which can therefore be also cleaned up at the same time. A predetermined size limit could be used to determine when items need cleanup.

I would be happy to work on it; since it involves a substantial amount of work, however, this activity needs to be planned for by the RP team. /cc @jsuereth

adriaanm commented 9 years ago

Ok, thanks for elaborating, @cunei -- we can work around on our end for now, but this would be good to have eventually.

SethTisue commented 1 year ago

our Jenkins behemoths occasionally run out of disk space, but not that often, and when it happens, it's because of a huge amount of temporary files that go away at the end of the run, so the next run always succeeds

we do eventually run out of inodes, but it takes a while

thus, I don't think change any further change in this area is likely