search

GoogleCloudPlatform / gcs-fuse-csi-driver

The Google Cloud Storage FUSE Container Storage Interface (CSI) Plugin.
Apache License 2.0
121 stars 30 forks source link

gcsfuse gcs-fuse-csi-driver increase default memory capacity #372

Open raed114 opened 1 week ago

raed114 commented 1 week ago

What you would like to accomplish: Increasing the default amount of the GCSFuse sidecar's memory from 100Mi to 200Mi instead of using a mutation webhook.

How this should work: Implemented either automatically and/or will have a native option to increase the memory of the sidecar container.*

Explanation of the problem: The gcs-fuse-csi-driver sidecar container seems to be repeatedly restarting and shows as OOMKilled all the while the node doesn't seem to run out of resources. or doesn't have any memory pressure issues. While the restart of this container eventually happened, it took 3 hours in this case (instead of a few minutes), which made the workload unresponsive, it became stuck until the eventual eviction of the pod

Several error messages after a deep investigation into it:

  1. MountVolume.SetUp failed for volume "<VOLUME_NAME>" : kubernetes.io/csi: mounter.SetUpAt failed to determine if the node service has VOLUME_MOUNT_GROUP capability: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/lib/kubelet/plugins/gcsfuse.csi.storage.gke.io/csi.sock: connect: connection refused"
  2. (combined from similar events): Memory cgroup out of memory: Killed process 2960932 (gcs-fuse-csi-dr) total-vm:1340248kB, anon-rss:100796kB, file-rss:26844kB, shmem-rss:0kB, UID:0 pgtables:372kB oom_score_adj:-997

Since this sidecar is created automatically and is managed by default, problems like these can cause serious downtime for the workload.

hime commented 1 day ago

Hi @raed114,

The connection refused error tends to be related to unavailability from the gcs-fuse-csi-driver, which makes me suspect there's something deeper going on. Could you provide the following?