Lots of NewRelic disk space errors

openaustralia / infrastructure

Automated setup and configuration for most of OpenAustralia Foundation's servers

8 stars 2 forks source link

Lots of NewRelic disk space errors #130

Closed jamezpolley closed 5 years ago

jamezpolley commented 5 years ago

Several newrelic services are reporting disk space issues. All of them are non-urgent, but they should be cleaned up so we notice when real issues crop up.

Cleaning up might involve changing threshhoulds, turning on cleanup jobs, or increasing disk space.

jamezpolley commented 5 years ago

"Disk full" alert set to warning at 70% for 10 minutes, critical 85% for 10 minutes. Lifted to 75%/60m and 90%/60m

oaf.org.au - / - apt autoremove got rid of 2Gb from /usr, primarily source of old kernel versions.
openaustralia.org - biggest user of disk space is old backups. Configured duply to automatically purge backups more than six months old each time the cron job runs.

jamezpolley commented 5 years ago

Used ansible to run apt autoremove on all hosts in EC2, which cleared up most other alerts.
Extended RTK's /data by 10Gb