search

geobeyond / Arpav-PPCV-backend

Backend di Piattaforma Proiezioni Climatiche per il Nord-Est.
Creative Commons Attribution 4.0 International
0 stars 1 forks source link

prevent staging disk from getting full of docker images #178

Closed ricardogsilva closed 2 weeks ago

ricardogsilva commented 1 month ago

Let's prevent the staging env's storage being full of old docker images. Docker has the command:

docker system prune --all

which allows removing unused stuff and free up space.

However, it would seem that our docker images should not occupy that much space, as they would be sharing much the same layers - investigate why this is not the case. As a last resort, maybe setting up a cronjob to run the aforementioned docker command would be in order.

ricardogsilva commented 3 weeks ago

After reading up on the docker storage drivers:

I checked that the stating env is indeed using the reccommended overlay2 driver:

Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false

Also read about how to reclaim disk space from:

https://www.virtualizationhowto.com/2023/11/docker-overlay2-cleanup-5-ways-to-reclaim-disk-space/

Finally, I decided to create a systemd timer, which is a replacement for cron on modern systems, to manage periodically calling docker system prune --all, which is the command to recover docker-related disk space.

About systemd timers: https://www.howtogeek.com/replace-cron-jobs-with-systemd-timers/

As per the above article:

The service file has the following contents:

# /etc/systemd/system/clean-docker-images.service
[Unit]
Description="Clean up docker images"

[Service]
Type=simple
ExecStart=/usr/bin/docker system prune --all
User=arpav

The timer file has the following contents:

# /etc/systemd/system/clean-docker-images.timer
[Unit]
Description="Timer for the clean-docker-images service"

[Timer]
Unit=clean-docker-images.service
OnCalendar=Sun *-*-* 01:00:00

[Install]
WantedBy=timers.target

As can be seen above, the timer is set to execute every Sunday at around 01h.

The timer is now enabled and started in the staging env, as reported by:

sudo systemctl status clean-docker-images.timer 
● clean-docker-images.timer - "Timer for the clean-docker-images service"
     Loaded: loaded (/etc/systemd/system/clean-docker-images.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Tue 2024-08-20 15:28:09 UTC; 2s ago
    Trigger: Sun 2024-08-25 01:00:00 UTC; 4 days left
   Triggers: ● clean-docker-images.service
ricardogsilva commented 3 weeks ago

@francbartoli if you agree, we can close this as completed, as I have already made the changes described above in the staging env