[3] Enabled fluentd-wc logging to S3

elastisys / compliantkubernetes-apps

Elastisys Compliant Kubernetes is an open source, Certified Kubernetes distribution designed according to the ISO27001 controls: providing you with security tooling and observability from day one.

https://elastisys.io/compliantkubernetes/

Apache License 2.0

46 stars 7 forks source link

[3] Enabled fluentd-wc logging to S3 #319

Open cristiklein opened 3 years ago

cristiklein commented 3 years ago

Is your feature request related to a problem? Please describe.

To add an additional layer of DR, but also enable alternative logging solutions, please make it possible to configure fluentd-wc so it logs to an S3 bucket. The access keys to said S3 bucket should be different from those used in the SC.

Describe the solution you'd like

Some configuration to enable fluentd-wc to log to S3, specifically the bucket endpoint and the API keys.

Definition of done:

I can configure the WC to log to an S3 bucket. (Disabled by default.)
Said logging works even if the SC is down.

robinelastisys commented 3 years ago

I turn this task over to someone else as it turned out to be over my head as a new DevOps. I made a few conclusions doe:

You cannot just copy the fluentd setup in SC to the WC-cluster without having to do major changes. To edit and develop the already existing fluentd-deamon in SC could be the way to go. To add the needed aws-secret to the WC-cluster, it can be set up in deploy-wc.sh.

tordsson commented 3 years ago

Let's start by figure out buffering capabilities of fluentd, to understand how important this task is.

OlleLarsson commented 3 years ago

Fluentd seems to be able to buffer up to 64GB https://docs.fluentd.org/configuration/buffer-section#buffering-parameters see, "total_limit_size". Not sure what we have it set to right now.

cristiklein commented 3 years ago

I couldn't find evidence for us overriding the default, so I strongly believe we are using the default. However, since we use hostPath, we run the risk of filling the host disks, which I usually set to 50GB.

How much disk space do we usually allocate on the disk?
Do we have alerts when host disk is running low?