search

elastisys / compliantkubernetes-apps

Elastisys Compliant Kubernetes is an open source, Certified Kubernetes distribution designed according to the ISO27001 controls: providing you with security tooling and observability from day one.
https://elastisys.io/compliantkubernetes/
Apache License 2.0
44 stars 7 forks source link

Apply chaos engineering on node-local-dns #1955

Open aarnq opened 6 months ago

aarnq commented 6 months ago

Concept to investigate

We've seen that node-local-dns can be highly impacted when there are network issues, and in its current state it can easily start to use a lot of memory that may have knock on effects causing nodes to go OOM. (From ~20MiB to over 1GiB has been seen in the wild.)

So, we should apply some chaos engineering and test different configurations that can better handle network issues better.

Artefacts to capture

Updated node-local-dns configuration based on chaos experiments.

Additional context

No response

davidumea commented 5 months ago

Would also be worth investigating if lowering the max_concurrent setting in coredns has any effect on the node-local-dns usage spikes.