Zuul logs should be rotated

SovereignCloudStack / issues

This repository is used for issues that are cross-repository or not bound to a specific repository.

https://github.com/orgs/SovereignCloudStack/projects/6

2 stars 1 forks source link

Zuul logs should be rotated #664

Open o-otte opened 2 weeks ago

o-otte commented 2 weeks ago

Lately, it happened that Zuul filled the VM Discs with a lot of Logs, resulting in a malfunctioning instance.

Hence, logs should be rotated.

Definition of Ready:

[ ] User Story is small enough to be finished within one sprint
[ ] User Story is clear and understood by the whole team
[ ] Acceptance criteria are defined
[ ] Acceptance criteria are clear and understood by the whole team

Definition of Done:

[ ] Logs in /var/log/zuul and /var/log/nodepool are handled with logrotate
[ ] Docker logs are set up only save a reasonable amout of log data
[ ] All acceptance criteria are met
[ ] Changes have been reviewed
[ ] CI tests have run successfully
[ ] Documentation has been updated
[ ] Release Notes have been updated

PixelPhantomX commented 2 weeks ago

Added logrotate for nodepool and zuul log volumes and limited log files on instances. This has been applied to the dev install playbook and tested.

PixelPhantomX commented 2 weeks ago

The change didn't work as expected within docker, so I have updated the docker-compose file to limit the files in that. Manually adjusted the same on prod, but it will not be affected until a restart is carried out, and added logrotate for the local Zuul and Nodepool logs on prod.

berendt commented 2 weeks ago

If you changed the log rotation behaviour in the Docker configuration I think the container has to be re-created.

https://docs.docker.com/config/containers/logging/configure/:

Restart Docker for the changes to take effect for newly created containers. Existing containers don't use the new logging configuration automatically.

PixelPhantomX commented 2 weeks ago

This is on prod and there is no maintenance planned for it, so not restarting them. The issue is low impact, for now, and the main problem has been addressed with the logrotate on the VM's filesystem.