search

metal-stack / metal-roles

Ansible roles for deploying metal-stack
MIT License
2 stars 2 forks source link

Docker Logs might consume all disk space on Switches #285

Closed vknabel closed 2 months ago

vknabel commented 3 months ago

After some time running the grafana/promtail container logs consume all free space on the disk ultimately resulting in failing switch syncs. It would make sense to limit the amount of logs to prevent this.

majst01 commented 3 months ago

Can you identify what exactly must be done to restrict the log volume ?

vknabel commented 3 months ago

Not sure how, but we need to set the options in https://docs.docker.com/config/containers/logging/json-file/

majst01 commented 3 months ago

Not sure how, but we need to set the options in https://docs.docker.com/config/containers/logging/json-file/

This applies to all other containers on the switch then, maybe there is some SONiC documentation on this topic

majst01 commented 2 months ago

Be aware that this affects all switches and management servers with the consequence that all daemons which are started with docker will be restarted

vknabel commented 2 months ago

With the solution all daemons started by the systemd-docker-service role will be restarted. If understand it correctly this does not apply sonic's daemons only 'ours'.