search

actions / runner-images

GitHub Actions runner images
MIT License
10.14k stars 3.05k forks source link

`No space left on device` regression on `ubuntu-latest` since February 5th #9344

Closed tomaarsen closed 8 months ago

tomaarsen commented 8 months ago

Description

(See also https://github.com/actions/runner-images/discussions/9329)

As of February 5th, I started noticing that my ubuntu-latest tests would start failing due to System.IO.IOException: No space left on device, while my windows-latest tests still consistently pass.

See here the first test failure with that error. I'm unsure how much space my CI tests actually use up, but I suspect that it's between 18 and 20GB. According to the GitHub docs here, the ubuntu-runner should have 150GB of space. I recognize that this is reduced by all of the installed tools, but one way or another, something changed on ~February 5th that seemingly caused the runners to have less space available.

Could it be that I am accidentally being given runners for private repositories? Those have 14GB of space, which would indeed not be enough.

It seems like there was indeed a release on February 5th, which may have caused this discrepancy. I would love some help to get my CI to pass again!

Platforms affected

Runner images affected

Image version and build link

20240201.1.0

https://github.com/UKPLab/sentence-transformers/actions/runs/7780698356

Is it regression?

20240126.1.0

Expected behavior

(I can't provide a link to the latest successful build in the last question, so I'll add it here: https://github.com/UKPLab/sentence-transformers/actions/runs/7752303635/job/21141574130)

I would expect for the CI to run the Python tests as instructed.

Actual behavior

When downloading ML models for testing, the runner encounters System.IO.IOException: No space left on device. I'm using the Standard GitHub hosted runners, and my repository is public, so according to the docs I should have ~150GB of disk space, just like on Windows. However, on Windows the runner works just fine.

I think the CI likely uses 18-20GB of disk space. Note that this started happening without changes to the code - all PRs & branches just started failing when the runner images updated.

Repro steps

Any CI triggered on the Sentence Transformers repository will trigger the failure.

mikhailkoliada commented 8 months ago

Hey! I guess we always guaranteed only 14GB of the space, there were some mistakes I the docs which are now fixed, screenshot taken from the link you provided. Sorry for the confusion.

image
tomaarsen commented 8 months ago

Oh, I see! That clears it up. I'll try to experiment with some solutions on my side, then.

tomaarsen commented 8 months ago

But you did reduce the disk size of the Ubuntu runners, right?

t1m0thyj commented 8 months ago

Hey! I guess we always guaranteed only 14GB of the space, there were some mistakes I the docs which are now fixed, screenshot taken from the link you provided. Sorry for the confusion.

@mikhailkoliada Thanks for clarification and updating the docs! Should this blog from a month ago be updated? Currently it is misleading because it says GitHub is offering "10 times the storage for their builds, for free".