search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.46k stars 2.21k forks source link

No space left on device #11575

Open kroeckx opened 8 months ago

kroeckx commented 8 months ago

In OpenSSL CI, we're regularly getting errors because of "No space left on device". An example log: https://github.com/openssl/openssl/actions/runs/7811610255/job/21307017321?pr=23502

t8m commented 8 months ago

I believe the main reason is that even that we run the CIFuzz on each and every pull request the oss-fuzz project is set up in such way that it builds and runs fuzzers on all the active release branches of OpenSSL.

I understand that doing so is useful for the OSS-Fuzz runs but it is definitely not for our GH CI CIFuzz workflow. We need to somehow fix this but I have no idea how to do so. I.e., if the Dockerfile and build.sh can have some conditionals that would skip cloning and building the branches and just build and run against the GH commit from the pull request.

jonathanmetzman commented 7 months ago

Sorry for the late reply. Is this still a problem for OpenSSL? The latest actions seem passing?

t8m commented 7 months ago

Yeah, CIFuzz CI runs are passing currently.

bernd-edlinger commented 7 months ago

Hmm, yeah, that was fixed by my recent PR #11618 That saves a lot of disk space by removing build artefacts that are no longer needed any more. Now the dist usage stays in the range 80-90% for each build step. Meanwhile I found the reason why this issue started: The github actions runner have changed their configuration around 6th of February, I can see that by looking at old CI runs on oss-fuzz: new runs: e.g. https://github.com/google/oss-fuzz/actions/runs/7800673195/job/21274044506

2024-02-06T13:46:04.3265761Z Filesystem      Size  Used Avail Use% Mounted on
2024-02-06T13:46:04.3266631Z /dev/root        73G   53G   20G  73% /
2024-02-06T13:46:04.3267091Z tmpfs           7.9G  172K  7.9G   1% /dev/shm
2024-02-06T13:46:04.3267557Z tmpfs           3.2G  1.1M  3.2G   1% /run
2024-02-06T13:46:04.3268121Z tmpfs           5.0M     0  5.0M   0% /run/lock
2024-02-06T13:46:04.3268787Z /dev/sdb15      105M  6.1M   99M   6% /boot/efi
2024-02-06T13:46:04.3269272Z /dev/sda1        74G  4.1G   66G   6% /mnt
2024-02-06T13:46:04.3269815Z tmpfs           1.6G   12K  1.6G   1% /run/user/1001

old runs: e.g. https://github.com/google/oss-fuzz/actions/runs/7761860320/job/21171166951

2024-02-02T21:42:14.9390652Z Filesystem      Size  Used Avail Use% Mounted on
2024-02-02T21:42:14.9391571Z /dev/root        84G   54G   31G  64% /
2024-02-02T21:42:14.9392692Z tmpfs           7.9G  172K  7.9G   1% /dev/shm
2024-02-02T21:42:14.9393689Z tmpfs           3.2G  1.1M  3.2G   1% /run
2024-02-02T21:42:14.9394754Z tmpfs           5.0M     0  5.0M   0% /run/lock
2024-02-02T21:42:14.9395639Z /dev/sdb15      105M  6.1M   99M   6% /boot/efi
2024-02-02T21:42:14.9396363Z /dev/sda1        63G  4.1G   56G   7% /mnt
2024-02-02T21:42:14.9397080Z tmpfs           1.6G   12K  1.6G   1% /run/user/1001
bernd-edlinger commented 7 months ago

but the /mnt folder only holds a swapfile and I dont see any reason why that volume size had to be increased at all.

ligurio commented 6 months ago

Seems you are using clusterfuzzlite on Github free runners. If it is so, you can cleanup space on runner before running fuzzing. I recently encountered the same problem and here is a commit ^1 with a fix. TL;DR: remove unused preinstalled packages.

catenacyber commented 6 months ago

CIFuzz has been disabled for Suricata due to this problem cf https://github.com/google/oss-fuzz/pull/11797 to try to get it back

bernd-edlinger commented 6 months ago

You can also look if this Cleanup step here helps: https://github.com/bernd-edlinger/openssl/commits/debug_cifuzz_workflow/

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       76026616 58037916  17972316  77% /
tmpfs            8182296      172   8182124   1% /dev/shm
tmpfs            3272920     1112   3271808   1% /run
tmpfs               5120        0      5120   0% /run/lock
/dev/sdb15        106832     6186    100646   6% /boot/efi
/dev/sda1       76829444  4194336  68686668   6% /mnt
tmpfs            1636456       12   1636444   1% /run/user/1001

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

Filesystem     1K-blocks     Used Available Use% Mounted on
/dev/root       76026616 36810956  39199276  49% /
tmpfs            8182296      172   8182124   1% /dev/shm
tmpfs            3272920     1112   3271808   1% /run
tmpfs               5120        0      5120   0% /run/lock
/dev/sdb15        106832     6186    100646   6% /boot/efi
/dev/sda1       76829444  4194336  68686668   6% /mnt
tmpfs            1636456       12   1636444   1% /run/user/1001

It delayed the workflow for a minute or so to delete all this useless stuff, but maybe that is also an alternative way out of the current situation.

catenacyber commented 6 months ago

Thanks Bernd, trying in https://github.com/OISF/suricata/pull/10816