ricardogsilva
commented
3 weeks ago After reading the docker docs on logging drivers I have decided to switch the staging env's default of using the docker json-file driver to the journald driver.
Compared to json-file, the journald driver performs automatic log rotation and is integrated with systemd.
I made relevant changes to the /etc/docker/daemon.json and restarted the docker daemon. I also recreated the containers of our staging env's stack so that they pick up this new config.
The staging env is thus now using the journald driver for storing logs.
Reading logs
Logs can now be read by either using the normal docker logs command, as usual or by using journalctl like this:
sudo journalctl -f CONTAINER_NAME=arpav-ppcv-staging-webapp-1(Omit the -f flag if you don't want to follow the logs)
Managing journald
These seem to be good basic references:
- https://www.digitalocean.com/community/tutorials/how-to-use-journalctl-to-view-and-manipulate-systemd-logs#journal-maintenance
- https://www.man7.org/linux/man-pages/man1/journalctl.1.html
In a nutshell:
# show space currently occupied with logs
sudo journalctl --disk-usage
# delete logs older than
sudo journalctl --vacuum-time=1years
# delete logs up to a size of
sudo journalctl --vacuum-size=1G
A note about the max size occupied by the journal:
SystemMaxUse= and RuntimeMaxUse= control how much disk space the journal may use up at most. SystemKeepFree= and RuntimeKeepFree= control how much disk space systemd-journald shall leave free for other uses. systemd-journald will respect both limits and use the smaller of the two values.
The first pair defaults to 10% and the second to 15% of the size of the respective file system, but each value is capped to 4G.
This means that on our staging server, the default journald config, which I did not change, would take up at maximum 4G.
Let's provide an appropriate log configuration to the staging compose file. The goal is to keep the server from filling up with logs