Extend docuum to work with volumes as well #68

Open savitojs opened 4 years ago

Description Currently, docuum only works with image threshold values. This approach can be extended to volumes as well. This is being driven from comment on GitLab Runners

Alternatives considered Everyone has their own alternatives mostly to run shell scripts in cron to run periodically else disable_cache to true for runners.

Additional context

docker volume inspect $volume has CreatedAt tag which can be used to evaluate age or docker system df to evaluate currently used space. Shell function for volumes mentioned in comment

PS: docker system df can be expensive 1 2

Thank you for this feature request and for bringing this to my attention. I was not aware that gitlab-runner uses volumes for caching purposes. I can imagine this feature being added to Docuum (especially if someone volunteers to contribute it), but I'm wondering how common this use case is. My understanding is that volumes are typically used for persistent storage, whereas images are considered more ephemeral, as they can be easily rebuilt when needed. There is a complexity and maintenance cost to introducing new functionality, so I just want to be sure it makes sense for Docuum to take that on. Docuum has kind of a "do one thing well" aesthetic right now.

For now, I will leave this issue open to indicate that this feature is within the realm of possibility. If someone has the bandwidth to implement it, I encourage them to discuss it with me first to align on the design.

Feel free to continue the discussion here.

Thanks, @stepchowfun for your comment, I understand that. I am not too sure if docuum is the right place to implement this solution but AFAIK many folks are complaining about this request (kind of FIFO removal of volumes). By adding this feature, the impact can be bigger in the adoption of this tool. At this time, One has to run docuum as well as custom scripts to run the cleanup based upon threshold values for volumes and using docker system prune --volumes

IMO the reason docuum exists is to fill a gap... that being that there is no easy way to keep track of when an image was last used because docker doesn't track this information. I would happily (and probably preferably) do this with a cron job if it was a feasible thing to do there.. but there is nothing to inspect to do this.. the closest thing is the image creation date but that doesn't mean anything with regards to actual image use (people tend to be happy using old images, even if it's not a great practice in theory).

Also consider: A) pruning volumes could be a destructive and cause data loss for the user.. must be very careful they understand what they are doing. B) gitlab-runner already includes a script in the package for cleaning up the runner cache volumes that it generates in /usr/share/gitlab-runner/clear-docker-cache C) it can easily be done by inspecting mtime on the volume files themselves:

@savitojs if it helps:

# Remove detached volumes older than 5 days

find /var/lib/docker/volumes -maxdepth 1 ! -path /var/lib/docker/volumes -type d -mtime +5 -printf ' %f' \
  | xargs -r /usr/bin/docker volume rm

Thank you @skyscooby. We are anyway using period job to do cleanup the unused. I was wondering if this fits within the tool. Thank you again.

I would like docuum to handle volumes as well (I'm also a Gitlab user). Otherwise I need to have a cronjob for the volume cleanup script and a docuum container for the images. Multiply that times hundreds of runner machines :)

I think it makes sense to keep this issue open. It seems like there is interest. I think it would be nice if it was an optional feature. Maybe it could also clean up old containers and networks so it could fully replace docker system prune.

Cool, thanks for reopening! - I mentioned this issue over on a related Gitlab issue: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27332.

This is interesting for my teams use case as well.

stepchowfun / docuum

Extend docuum to work with volumes as well #68