search

yugabyte / yugabyte-db

YugabyteDB - the cloud native distributed SQL database for mission-critical applications.
https://www.yugabyte.com
Other
8.93k stars 1.06k forks source link

[docdb] Better log rotation #3604

Open ddorian opened 4 years ago

ddorian commented 4 years ago

Jira Link: DB-2438 Hi

We need a better log rotation policy. 1 way is to keep static naming and allow another process to rotate: https://github.com/google/glog/pull/19 (patch https://github.com/google/glog/commit/58d7f873dcde8cf0fe6f37817643a7ff4e49e6d4)

Another way would be to add an option glog_delete_after and a simple bg thread to enforce the policy. For now, having glog_delete_after=<size> should be a good start. It will take us a day or two to implement and probably save us devops trouble in different environments (k8s sidecar comes to mind).

bmatican commented 4 years ago

Opening this up for now, as the bg thread + size flag should not be too complicated.

iSignal commented 4 years ago

We hit this recently in a customer scenario, where glogs took up 20G space over time. We need to be able to have the yb-tserver process automatically cap its logs when run in some of these scenarios where having a separate log prune script is overkill.

ddorian commented 4 years ago

Example: I have a local installation that I've used for a long time. It had 85GB logs in 1 yb-tserver.

tylarb commented 3 years ago

I like the idea of static naming followed by log rotation using an external tool - this allows easy transport of logs to syslog or other OS managed logging systems.

tylarb commented 3 years ago

This feature could be relevant for k8s as well - where we let k8s control the logs size, rather than having a sidecare process / container that only does log rotation.

kulminaator commented 3 years ago

I'm also interested in seeing a more predictable log rotation scheme.

From the top of the hill that i sit on - the hourly or daily rotation scheme that postgres uses makes it very predictable how and where stuff is logged (and also allows easy instructions for external tooling how they can archive - ship - delete log files).

Current scheme where my ERROR log files hardly get any content they will never reach any rotation at all.