Open Christoph-AK opened 2 years ago
github-actions[bot]
commented
2 years ago Hi there! 👋🏼 As you're new to this repo, we'd like to suggest that you read our code of conduct as well as our contribution guidelines. Thanks a bunch for opening your first issue! 🙏
MrYutz
commented
2 years ago This was also an issue for us last week on the same set of specs. AWS Ubuntu with --cleanup enabled did not actually prune the disk, causing an upgrade to fail. Still digging to make sure the upgrade failed due to only disk space, but there were a bunch of previous upgrades that had not been pruned by watchtower.
pewter77
commented
2 years ago Seems like this was an issue before here: https://github.com/containrrr/watchtower/issues/662 might be related. I'm having this issue. Filled up to 90+GB. I only update using this on a weekly basis though so I imagine this is a huge annoyance for others.
piksel
commented
2 years ago Could someone show the logs of a run? If you are running with --cleanup there should be a log entry for every removed image.
pewter77
commented
2 years ago services:
watchtower:
image: containrrr/watchtower
container_name: watchtower
hostname: watchtower
environment:
- TZ=${TZ}
- WATCHTOWER_CLEANUP=true
- WATCHTOWER_ROLLING_RESTART=true
#- WATCHTOWER_DEBUG=true
- WATCHTOWER_SCHEDULE=* 2 6 * * 5
#- WATCHTOWER_RUN_ONCE=true
## Watchtower autodetects discord urls and uses shoutrr##
- WATCHTOWER_NOTIFICATIONS=slack
- WATCHTOWER_NOTIFICATION_SLACK_HOOK_URL=${WATCHTOWER_WEBHOOK_URL}/slack
- WATCHTOWER_NOTIFICATION_SLACK_IDENTIFIER=*removed*
- WATCHTOWER_NOTIFICATION_SLACK_ICON_EMOJI=":whale:"
volumes:
- /var/run/docker.sock:/var/run/docker.sock
restart: alwaysNot sure if this helps, but here's my logs and my compose file. The images do get uploaded, the containers do restart. But old images aren't removed.
Christoph-AK
commented
2 years ago So far I only got the behavior on one production system. As it is important that that one runs well, I included the gc container into my compose.
gc:
image: drone/gc
restart: unless-stopped
container_name: gc
volumes:
- /var/run/docker.sock:/var/run/docker.sock
environment:
- GC_INTERVAL=1hWith that all images get cleaned as expected.
Unfortunately with that I can't really provide any useful logs on this.
The continuos log of watchtower: https://gist.github.com/Christoph-AK/052523cf83c093a3cba2dca63684c7b3
Thanks for looking into this!
piksel
commented
2 years ago @pewter77 so, the containers tika, gotenberg and portainer were updated, but their previous images (3a8c80c1ea4c, ba3c7d7ad873 and 1528f5b692fe respectively) were not removed?
The fact that there are 10-21 containers that seem to just disappear during the check also seems concerning...
Update: Perhaps the reason for your storage filling up is because of all those failed updates? For every check it would pull the new image, but since it cannot update the container it won't remove the old one.
piksel
commented
2 years ago @Christoph-AK Could you check what images are present in docker images and compare that to the images removed in the log?
Watchtower doesn't run system prune, it just tries to remove the old image when a new one was updated to.
Also, I added some more logging to #1466 so that all images that are removed (including parent images) are actually logged (using --debug).
pewter77
commented
2 years ago _watchtower_logs (1).txt So watchtower updated to 1.5.1 and I got more logs, it seems there's some weird logic where it shuts down some containers, shows that container doesn't exist anymore and then doesn't finish deleting it? Again all my containers are up and running, but many different images are still there unused.
I should mention that I do have connectivity issues because of my location, I have to use obfuscating proxies and such to pass over country wide internet firewalls. Though they are fairly consistent, sometimes they timeout. But like I said these containers seem to be updated, just the leftover images are still here so I'm not sure this is the issue.
Describe the bug After running watchtower with cleanup for a while, my /var/lib/docker/overlay2 directory filled up and wasn't cleaned.
I needed to run
docker system prunefor the space to be freed.To Reproduce Run watchtower with cleanup for a while with a frequently updating service, like grafana-oss.
Expected behavior The space of images gets correctly cleaned up after updates.
Environment Ubuntu on AWS, Docker version 20.10.14, build a224086