Open cristiklein opened 3 years ago
I turn this task over to someone else as it turned out to be over my head as a new DevOps. I made a few conclusions doe:
You cannot just copy the fluentd setup in SC to the WC-cluster without having to do major changes. To edit and develop the already existing fluentd-deamon in SC could be the way to go. To add the needed aws-secret to the WC-cluster, it can be set up in deploy-wc.sh.
Let's start by figure out buffering capabilities of fluentd, to understand how important this task is.
Fluentd seems to be able to buffer up to 64GB https://docs.fluentd.org/configuration/buffer-section#buffering-parameters see, "total_limit_size". Not sure what we have it set to right now.
I couldn't find evidence for us overriding the default, so I strongly believe we are using the default. However, since we use hostPath, we run the risk of filling the host disks, which I usually set to 50GB.
Is your feature request related to a problem? Please describe.
To add an additional layer of DR, but also enable alternative logging solutions, please make it possible to configure fluentd-wc so it logs to an S3 bucket. The access keys to said S3 bucket should be different from those used in the SC.
Describe the solution you'd like
Some configuration to enable fluentd-wc to log to S3, specifically the bucket endpoint and the API keys.
Definition of done: