search

mozmeao / infra

Mozilla Marketing Engineering and Operations Infrastructure
https://mozilla.github.io/meao/
Mozilla Public License 2.0
59 stars 12 forks source link

Add disk space cleanup cronjob to gitlab runners #1313

Closed duallain closed 4 years ago

duallain commented 4 years ago

Docker objects filled our disk, let's add a cronjob that does cleanup.

docker system prune --volumes --force

Should work. Maybe run it hourly with a filter (since it's incremental, and running more frequently should make it run faster, with smaller resource spikes) of 24hr? --filter "until=24h".

glogiotatidis commented 4 years ago

There's already a cron job, but we probably need to run it more often (with the filter command too, nice tip) and also increase the disk space. https://github.com/mozmeao/infra-services/blob/master/gitlab-runners/packer/ansible/playbook.yml#L98

glogiotatidis commented 4 years ago

FTR 

root@ip-172-31-31-72:/home/admin# docker system prune --volumes --force --filter 'until=12h'
ERROR: The "until" filter is not supported with "--volumes"
glogiotatidis commented 4 years ago

https://github.com/mozmeao/infra-services/pull/51

glogiotatidis commented 4 years ago

Updated here https://github.com/mozmeao/infra-services/pull/54 and applied