s3-plugin exit code 2

[tomahkvt]

/kind bug

NOTE: If this is a filesystem related bug, please take a look at the Mountpoint repo to submit a bug report

What happened? We see quite often s3-csi-node- pods' stop with exit code 2.

2024-01-30 13:54:45.207 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:54:45.207 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:54:46.075 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:54:46.075 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:54:46.082 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:54:46.082 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-rwzj5
2024-01-30 13:55:35.781 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 13:55:35.781 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 13:55:36.435 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 13:55:36.435 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 13:55:36.522 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 13:55:36.523 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-xzlwh
2024-01-30 14:14:12.884 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:14:12.884 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:14:13.639 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:14:13.639 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:14:13.647 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:14:13.647 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-hndkx
2024-01-30 14:24:42.846 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-56jpl
2024-01-30 14:24:42.846 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-56jpl
2024-01-30 14:24:43.675 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-56jpl
2024-01-30 14:24:43.675 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-56jpl
2024-01-30 14:24:43.790 
problem_container_name=liveness-probe exit_code=2 problem_pod_name=s3-csi-node-56jpl
2024-01-30 14:24:43.790 
problem_container_name=s3-plugin exit_code=2 problem_pod_name=s3-csi-node-56jpl

What you expected to happen? We want to understand the reason pods restart.

How to reproduce it (as minimally and precisely as possible)? We use ARM image public.ecr.aws/mountpoint-s3-csi-driver/aws-mountpoint-s3-csi-driver:v1.2.0 repoURL: "https://awslabs.github.io/mountpoint-s3-csi-driver" targetRevision: 1.2.0 chart: aws-mountpoint-s3-csi-driver

Anything else we need to know?: We have this message in the pods logs: Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,} Also, we noticed a memory leak

Environment

• Kubernetes version (use kubectl version): Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.27.8-eks-8cb36c9

• Driver version: -We use ARM image public.ecr.aws/mountpoint-s3-csi-driver/aws-mountpoint-s3-csi-driver:v1.2.0

[jjkr]

I am unable to reproduce this. Any more information you can provide here will be helpful, especially:

• What OS and version is your cluster running?
• Logs from the failed containers will be very helpful. A command similar to this should give logs from the crashed container: kubectl logs -f pod-name -n -p
• What the driver was doing at the time (what kind of buckets are you mounting, are you performing io, etc.)

[tomahkvt]

Hi @jjkr.

• OS uname -r 5.10.201-191.748.amzn2.aarch64
• I attached a file with logs to this answer
• I can't say that happens something special. I use the S3 Explore-logs-2024-02-05 17_30_12.txt CSI driver for low-load services.