Containers on test-equinix-ubuntu2004-docker-arm64-3 low on disk space

[richardlau]

Containers on test-equinix-ubuntu2004-docker-arm64-3 are being flagged at being 90% disk usage:

[richardlau]

root@test-equinix-ubuntu2004-docker-arm64-3:~# df -h /home
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3  871G  738G   90G  90% /
root@test-equinix-ubuntu2004-docker-arm64-3:~# du -hs /home/*
713G    /home/iojs
root@test-equinix-ubuntu2004-docker-arm64-3:~# du -hs /home/iojs/*
2.5M    /home/iojs/get-pip.py
270G    /home/iojs/test-equinix-debian11_container-armv7l-2
3.9G    /home/iojs/test-equinix-rhel8_container-arm64-2
3.4G    /home/iojs/test-equinix-ubuntu1804_container-arm64-2
589M    /home/iojs/test-equinix-ubuntu1804_sharedlibs_container-arm64-4
463M    /home/iojs/test-equinix-ubuntu1804_sharedlibs_container-arm64-5
441M    /home/iojs/test-equinix-ubuntu1804_sharedlibs_container-arm64-6
35G     /home/iojs/test-equinix-ubuntu2204_container-arm64-2
35G     /home/iojs/test-equinix-ubuntu2204_container-arm64-4
35G     /home/iojs/test-equinix-ubuntu2204_container-arm64-6
32G     /home/iojs/test-equinix-ubuntu2204_container-arm64-8
279G    /home/iojs/test-equinix-ubuntu2204_container-armv7l-2
root@test-equinix-ubuntu2004-docker-arm64-3:~#

The home directories for the armv7l containers taking 270G each would be the obvious thing to look at.

[richardlau]

root@test-equinix-ubuntu2004-docker-arm64-3:~# du -hs /home/iojs/*armv7l-2/build/workspace/node-test-binary-armv7l/.git
269G    /home/iojs/test-equinix-debian11_container-armv7l-2/build/workspace/node-test-binary-armv7l/.git
274G    /home/iojs/test-equinix-ubuntu2204_container-armv7l-2/build/workspace/node-test-binary-armv7l/.git
root@test-equinix-ubuntu2004-docker-arm64-3:~#

[richardlau]

I went round and ran

git gc

on all the node-test-binary-armv7l workspaces in the armv7l containers. For the really large ones I had to temporarily drop git's pack.threads config option to 1 (!) as otherwise git gc was running out of memory:

git config pack.threads 1

Once git gc completed successfully, I unset pack.threads and checked git gc was still able to run (with much less to pack).

git config --unset pack.threads

I've added a git gc at the end of node-test-binary-armv7l (we were already doing something similar for the compile job) -- hopefully doing the collection frequently will keep the size of the repository in check.

[richardlau]

I've added a git gc at the end of node-test-binary-armv7l (we were already doing something similar for the compile job) -- hopefully doing the collection frequently will keep the size of the repository in check.

FWIW, either side of https://ci.nodejs.org/job/node-test-binary-armv7l/RUN_SUBSET=native,nodes=ubuntu2204-armv7l/14770/console running:

ubuntu@test-osuosl-ubuntu2004-docker-arm64-1:/home/iojs$ sudo du -hs *
2.5M    get-pip.py
2.9G    test-osuosl-debian11_container-armv7l-1
3.7G    test-osuosl-rhel8_container-arm64-1
404M    test-osuosl-ubuntu1804_container-arm64-1
420M    test-osuosl-ubuntu1804_sharedlibs_container-arm64-1
3.3G    test-osuosl-ubuntu2204_container-arm64-1
3.3G    test-osuosl-ubuntu2204_container-arm64-2
11G     test-osuosl-ubuntu2204_container-armv7l-1
ubuntu@test-osuosl-ubuntu2004-docker-arm64-1:/home/iojs$ sudo du -hs *
2.5M    get-pip.py
2.9G    test-osuosl-debian11_container-armv7l-1
3.7G    test-osuosl-rhel8_container-arm64-1
404M    test-osuosl-ubuntu1804_container-arm64-1
420M    test-osuosl-ubuntu1804_sharedlibs_container-arm64-1
3.3G    test-osuosl-ubuntu2204_container-arm64-1
3.3G    test-osuosl-ubuntu2204_container-arm64-2
9.1G    test-osuosl-ubuntu2204_container-armv7l-1
ubuntu@test-osuosl-ubuntu2004-docker-arm64-1:/home/iojs$

which shows the workspace for test-osuosl-ubuntu2204_container-armv7l-1 shrunk after the job ran 🎉.