search

jenkins-infra / helpdesk

Open your Infrastructure related issues here for the Jenkins project
https://github.com/jenkins-infra/helpdesk/issues/new/choose
17 stars 10 forks source link

[trusted.ci.jenkins.io] alert `Disk space is below 1GB` #3491

Closed smerle33 closed 1 year ago

smerle33 commented 1 year ago

Service(s)

trusted.ci.jenkins.io

Summary

We received an alert : Disk space is below 1GB free i-ead37377

After some research it appear to be on trusted.ci

Filesystem   Size Used Avail Use% Mounted on
/dev/xvda1    16G  14G 1.1G 93% /

we did follow our runbook and cleaned the docker : docker image ls -q | xargs docker rmi docker system prune --volumes journalctl --vacuum-size=400M 

Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       16G  8.3G  6.6G  56% /

Reproduction steps

No response

smerle33 commented 1 year ago

we need to improve the datadog detection to get the name of the instance, it's cloud provider and role at first glance. this may help adding tags on datadog:

https://forge.puppet.com/modules/datadog/datadog_agent/readme#configuration-variables

smerle33 commented 1 year ago

During the research for information as to which controller was impacted, we discovered that ci.jenkins.io had the same kind of disk usage : 

/dev/sdb1    492G 443G  24G 95% /var/lib/jenkins

this need to be fixed.

dduportal commented 1 year ago