essh
commented
1 year ago I've managed to build a reliable reproduction for this issue that I can now share. A quick summary is that the impact seems to depend on instance type. I have been able to consistently reproduce this issue on c5d.xlarge & c5a.xlarge instance types (x86_64). I have seen some bpf_jit memory growth on c6g.xlarge instances (arm64) but it seems a bit slower and I haven't seen containers fail to create on these nodes yet as a result. I can't reproduce this issue on a t3a.large instance as bpf_jit memory levels remain pretty consistent.
The easiest way to reproduce this is to spin up a fresh EKS 1.24 cluster and add a single node of the required instance type (this makes it easier to observe) running EKS AMI v20230217 (or v20230304). Then run the following commands:
kubectl delete clusterrolebinding eks:podsecuritypolicy:authenticated
kubectl delete clusterrole eks:podsecuritypolicy:privileged
kubectl delete podsecuritypolicy eks.privileged
kubectl apply -f https://gist.githubusercontent.com/essh/f7dd219a48df25e7294847484da112b7/raw/503ff9a8f32f19430040cd65c213479979bfcc3c/bpf-jit-leak.yamlThis removes the eks.privileged PSP, installs PSPs that use seccomp and starts up a simple app with some exec probes that trigger the issue. The container used for this app is built from the source available at https://github.com/essh/grpc-greeter-node.
Once this is running you can observe memory growth by executing sudo cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}' on the node. You can tweak the replica count up and down to speed up/slow down this process. If you leave it long enough the value will exceed net.core.bpf_jit_limit and you will end up with failure to create containers/exec probes. We were seeing this after about 2-3 days with our node types/workloads in our environment.
This same test against EKS AMI v20230203 nodes or lower (kernel 5.4) does not exhibit this issue.
cartermckinnon
borkmann
stevo-f3
dims
q2ven
rarecrumb
SimonKO9
Wyifei
billo-zymantas
kirinnee
tomislater
mmerkes
alexku7
ChrisV78
dougbaber
mynkkmr
reedjosh
islishude
radimk
KrisJohnstone
mchlbataller
marcincuber
bla-ckbox
macmiranda
nille
What happened:
After upgrading EKS nodes from
v20230203tov20230217on our1.24EKS clusters after a few days a number of the nodes have containers stuck inContainerCreatingstate or liveness/readiness probes reporting the following error:This issue is very similar to https://github.com/awslabs/amazon-eks-ami/issues/1179. However, we had not been seeing this issue on previous AMIs and it only started to occur on
v20230217(following the upgrade from kernel 5.4 to 5.10) with no other changes to the underlying cluster or workloads.We tried the suggestions from that issue (
sysctl net.core.bpf_jit_limit=452534528) which helped to immediately allow containers to be created and probes to execute but after approximately a day the issue returned and the value returned bycat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'was steadily increasing.What you expected to happen:
ReadyHow to reproduce it (as minimally and precisely as possible):
I don't currently have a reproduction that I can share due to my current one using some internal code (I can hopefully produce a more generic one if required when I get a chance).
As a starting point we only noticed this happening on nodes that had pods scheduled on them which had an
execliveness & readiness probe running every 10 seconds that performs a health check against a gRPC service usinggrpcurl. In addition to this we also have a default Pod Security Policy (yes we know they are deprecated 😄) that has the following annotationseccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default.These two conditions seem to be enough to trigger this issue and the values reported by
cat /proc/vmallocinfo | grep bpf_jit | awk '{s+=$2} END {print s}'will steadily increase over time until containers can no longer be created on the node.Anything else we need to know?:
Environment:
aws eks describe-cluster --name <name> --query cluster.platformVersion):"eks.4"aws eks describe-cluster --name <name> --query cluster.version):"1.24"v20230217uname -a):5.10.165-143.735.amzn2.x86_64 #1 SMP Wed Jan 25 03:13:54 UTC 2023 x86_64 x86_64 x86_64 GNU/Linuxcat /etc/eks/releaseon a node):Official Guidance
Kubernetes pods using SECCOMP filtering on EKS optimized AMIs based on Linux Kernel version 5.10.x may get stuck in
ContainerCreatingstate or their liveness/readiness probes fail with the following error:When a process with SECCOMP filters creates a child process, the same filters are inherited and applied to the new process. The Amazon Linux kernel versions 5.10.x are affected by a memory leak that occurs when parent process is terminated while creating a child process. When the total amount of memory allocated for SECCOMP filter is over the limit, a process cannot create a new SECCOMP filter. As a result, the parent process fails to create a new child process and the above error message will be logged.
This issue is more likely to be encountered with kernel versions
kernel-5.10.176-157.645.amzn2andkernel-5.10.177-158.645.amzn2where the rate of the memory leak is higher.Amazon Linux will be releasing the fixed kernel by May 1st, 2023. We will be releasing a new set of EKS AMIs with the updated kernel latest by May 3rd, 2023.