adoptium / infrastructure

This repo contains all information about machine maintenance.
Apache License 2.0
86 stars 101 forks source link

Nagios: mitigate large workspaces on dockerhost-equinix-ubuntu2204-x64-1 #3405

Open sxa opened 8 months ago

sxa commented 8 months ago

This machine appears to have some large workspaces in some of the containers - we should look at these and see if there was an obvious reason why they are so big and haven't been deleted:

:warning: HOST: dockerhost-equinix-ubuntu2204-x64-1 SERVICE: Check Docker Container Health STATE: WARNING MESSAGE: WARNING - These Docker Containers Have Large Workspaces: ,ubi8.2226,0e592afe0a76 See Nagios

sxa commented 8 months ago

Each of these jobs have worksapces around the 4Gb mark:

drwxrwxr-x 7 jenkins jenkins 4096 Feb 20 12:24 Test_openjdk11_hs_sanity.system_x86-64_linux
drwxrwxr-x 7 jenkins jenkins 4096 Jan 30 03:39 Test_openjdk17_hs_extended.openjdk_x86-64_linux_rerun
drwxrwxr-x 6 jenkins jenkins 4096 Feb 13 00:56 Test_openjdk21_hs_extended.openjdk_x86-64_linux_rerun
sxa commented 8 months ago

Analysis:

sxa commented 8 months ago

@steelhead31 @smlambert FYI I've removed the rerun directories based on the above so the alerts should stop but we need to be mindful that we may have an issue with space leakage in the situation where we hit a timeout.