aws-efs-csi-driver efs-proxy OOM killed

[nan-coupa]

/kind bug

What happened?

efs-proxy OOM killed when the EKS node has plenty of memory.

What you expected to happen?

efs-proxy runs without issue.

How to reproduce it (as minimally and precisely as possible)?

Run container using EFS volume. The process efs-proxy gets OOM killed.

Anything else we need to know?:

Environment

• Kubernetes version (use kubectl version): 1.27
• Driver version: 2.0.0

Please also attach debug logs to help us better diagnose

[root@ip-10-0-12-209 /]# free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        912M        8.1G        6.1M        6.4G         14G
Swap:            0B          0B          0B

[root@ip-10-0-12-209 /]# ps -ef | grep efs-proxy
root        8538    6216  0 17:50 ?        00:00:00 [efs-proxy] <defunct>
root      199374    6236  1 19:24 ?        00:00:00 [efs-proxy] <defunct>

/var/log/messages indicate process oom killed and any attempt to restart csi-driver result in the same:

Jun 21 19:22:58 ip-10-0-12-209 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50beeb08_b8d5_460a_92f7_15010a01cd30.slice/cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50beeb08_b8d5_460a_92f7_15010a01cd30.slice/cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,task=efs-proxy,pid=197121,uid=0
Jun 21 19:22:58 ip-10-0-12-209 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,mems_allowed=0,oom_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50beeb08_b8d5_460a_92f7_15010a01cd30.slice/cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,task_memcg=/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50beeb08_b8d5_460a_92f7_15010a01cd30.slice/cri-containerd-ca0c308b91e6712a9f57d9c4eb300cc8e3acb80a2d3f342f6c858d4998bbb945.scope,task=efs-proxy,pid=197121,uid=0
Jun 21 19:22:58 ip-10-0-12-209 kernel: Memory cgroup out of memory: Killed process 197121 (efs-proxy) total-vm:266300kB, anon-rss:100688kB, file-rss:6336kB, shmem-rss:0kB, UID:0 pgtables:320kB oom_score_adj:-997
Jun 21 19:22:58 ip-10-0-12-209 kernel: Memory cgroup out of memory: Killed process 197121 (efs-proxy) total-vm:266300kB, anon-rss:100688kB, file-rss:6336kB, shmem-rss:0kB, UID:0 pgtables:320kB oom_score_adj:-997

• Instructions to gather debug logs can be found here

[seanzatzdev-amazon]

Does the issue persist on v2.0.4 of the driver? Previous versions had an issue with zombie efs-proxy processes.

Otherwise, do you have any memory or cpu limits on your driver?

[seanzatzdev-amazon]

Can you also send your storage class,pvc, and pod files?

[nan-coupa]

Further testing shows this works with v2.0.4.