search

faster-cpython / bench_runner

Code for running pyperformance benchmarks on Github Action runners
BSD 3-Clause "New" or "Revised" License
3 stars 5 forks source link

Self hosted runners running out of disk space #244

Open mdboom opened 1 month ago

mdboom commented 1 month ago

The pystats self-hosted runner ran out of disk space and therefore stopped fulfilling requests. While this issue is most acute on that runner, since it's a VM with only 30GB of disk space, these issues probably apply equally to all self-hosted runners, so we should have a solution that applies to both.

Here are some sources of disk space usage that should be addressed:

  1. The .git directory for the benchmarking repo. We keep the repo size down by periodically manually deleting old history. However, since the checkout is retained on the runners, this doesn't actually remove the old content. This was initially 12GB, and after running git gc it went to 4GB. This can be automated.

  2. There are multiple versions of the dependencies for Github Actions installed, left behind when GHA self-updates. These have the names externalsX.YY and are roughly .5GB each. There is an issue for this that claims a fix has been merged, but nonetheless these dirs seem to remain.

  3. The pip cache is about 1GB. Unfortunately, I don't think it's possible to control the amount of space used for the pip cache. The best we can probably do is periodically purge it.

mdboom commented 1 week ago

Step 1. has been implemented.

Step 2. requires an upstream fix from Github, IMHO.

Step 3. is still TBD.