search

metal-stack / os-metal-extension

gardener extension to handle userdata creation in the ignition format
3 stars 2 forks source link

log rotation inconsistent for docker / containerd #16

Open mwennrich opened 3 years ago

mwennrich commented 3 years ago

docker/daemon.json has:

  "log-opts": {
    "max-size": "50m",
    "max-file": "3"
  },

containerd-lograte.services uses /etc/systemd/containerd.conf

/var/log/pods/*/*/*.log {
    rotate 14
    copytruncate
    missingok
    notifempty
    compress
    maxsize 100M
    daily
    dateext
    dateformat -%Y%m%d-%s
    create 0644 root root
}
mwennrich commented 3 years ago

related to #14 (maxsize still for containerd-pod-logs is 100M instead of 50M, and keep is 14 instead of 3)

majst01 commented 3 years ago

I am unsure if we overwrite the configfile of containerd anywhere, i guess we should do that at the same place we modify/create the docker/daemon.json which is in metal-images.

mwennrich commented 3 years ago

docker/daemon.json gets created here: https://github.com/metal-stack/os-metal-extension/blob/master/pkg/internal/templates/cloud-init.sh.template#L48-L54

containerd/config.toml here: https://github.com/metal-stack/os-metal-extension/blob/master/pkg/ignition.go#L20-L24

but the content of /etc/systemd/containerd.conf seems to be hardcoded here: https://github.com/gardener/gardener/blob/v1.18.2/pkg/operation/botanist/extensions/operatingsystemconfig/original/components/logrotate/logrotate.go#L64-L75 :-/

mwennrich commented 3 years ago

Another difference between docker-based and containerd-based clusters regarding the logs:

with docker, the actual location of the logs is under /var/lib/:

shoot--test--fra-equ01-default-worker-5f9b5-ltk9g:/var/log/pods/monitoring_promtail-tzwf4_322e8e7e-35ed-4bd0-9ad9-7cc6492ba9b0/promtail# ls -lart
lrwxrwxrwx 1 root root  165 May 19 13:28 0.log -> /var/lib/docker/containers/e3f294b9dd5742730b6c1ffc60f57476c31291a4f7dfd55651ead9c0aa8979f6/e3f294b9dd5742730b6c1ffc60f57476c31291a4f7dfd55651ead9c0aa8979f6-json.log

with containerd, the logs are on the small /:

shoot--phjjbm--s3-test-01-group-0-77fc5-k5wp6:/var/log/pods/monitoring_promtail-n66cb_0fe3e17b-2ad6-44c0-9fc0-568633a73841/promtail# ls -lart
-rw-r----- 1 root root  6586 Jul  6 23:32 0.log-20210707-1625608800.gz
-rw-r----- 1 root root  4113 Jul  7 23:29 0.log-20210708-1625695200.gz
majst01 commented 3 years ago

Uh this is not good, thanks for the digging. Maybe one simple solution would be to put /var onto a separate partition with the help of fsl

Gerrit91 commented 8 months ago

This is by now handled and configurable by Gardener, I think we should not mess around with this?

https://github.com/gardener/gardener/blob/v1.87.2/docs/usage/logging.md#container-logs-rotation-and-retention