gce-pd-driver container OOM killed after upgrade to GKE 1.28

JordanP commented 4 months ago

Hi, You probably have not control over how the pd-csi daemonset is deployed on GKE, but I am taking my chance.

After upgrading to GKE 1.28, some gce-pd-driver containers started to be OOM killed. Before getting killed, the last log line is Checking for issues with fsck on disk: /dev/disk/by-id/google-restore-aus-southeast1-fcb9-pg-data-pg-main-0-7279. That disk is a multi TB disk attached to a pod. My guess is 50MB (resources.memory.limit set by GKE for that gce-pd-driver container) is not enough to run fsck on such a large disk.

Any chance you could reach to someone at GKE to increase that memory limit (although baseline usage, fsck excepted is ~10MB so 50MB seems reasonable) ? Or how could I skip that fsck check ?

If that helps, this is part of my Go code (running elsewhere) that seems to, down the line, trigger the call to fsck.

    // make the request to the api /metrics endpoint and handle the response
    req := clientset.
        CoreV1().
        RESTClient().
        Get().
        Resource("nodes").
        Name(nodeName).
        SubResource("proxy").
        Suffix("metrics")
    respBody, err := req.DoRaw(ctx)
    if err != nil {
        return errors.Errorf("failed to get stats from kubelet on node %s: with error %s", nodeName, err)
    }

Thanks !

(please, don't recommend I reach to my TAM at GCP, we don't have one haha)

JordanP commented 4 months ago

Actually I think there's more to it but I can't figure it out yet In the instance, it looks like only the spawned fsck process got SIGKILL but not the gce-pd-driver process, so it was able to make progress eventually.

JordanP commented 4 months ago

I am starting to feel that there's a deeper issue. Let's take a step back and explain the context. I have a script that, every 30min:

Create a new GCE disk from a disk snapshot

Attach the disk to a bare K8s pod

volumes=[
    V1Volume(
        name="pg-data",
        gce_persistent_disk=V1GCEPersistentDiskVolumeSource(fs_type="ext4", pd_name=disk_name),
    )
],

Do some stuff with the pod, for about 10min
Then the pod is deleted then the GCE disk is deleted.

In my GKE cluster, I have a monitoring solution called Datadog and it deploys (i think under the hood) kube-state-metrics

Let's see all the logs related to one of the GCE disk mentioned above (disk is named XYZ-pg-data-pg-main-0-6864)

Now, let's see all the logs for the csi pod

That script is running every 30min, on 10 GKE clusters. The OOM situation happens from time to time, like once per hour in total. That's why it feels like a race condition.

JordanP commented 4 months ago

I now think the key part is that I take a "GCE disk snapshot" and then restore a volume from that snapshot. When the GCE disk snapshot is performed, I guess the FS still have the "needs recovery" bit set (that bit, if I understand correctly, is unset when the FS is properly unmounted), so on next mount fsck will enter the recovering journal phase. And that needs a lot (more than the 50 MB mem limit set by GKE) of memory to succeed.

mattcary commented 4 months ago

The OOMs are a known issue, and we've been trying to fix them (eg, #1168). The fsck one is harder, though, and we are currently trying to find a solution.

The best way to increase the memory for a daemonset instance is to use a mutating webhook. Let me know if you need more details. There's no way for GKE to manually change the daemonset configuration as it turns out.

You've also run into another known sharp edge with snapshots -- they're hard to do online. To avoid neediing to recover the journal on the snapshot, you need either to do them offline (ie, after unmounting), or to quiesce & freeze the volume (which is no possible to do without being a privileged pod). This is a known problem (see https://github.com/kubernetes-csi/external-snapshotter/issues/566#issuecomment-893456072, for example), and unfortunately there's no clear path forward.

JordanP commented 4 months ago

The best way to increase the memory for a daemonset instance is to use a mutating webhook.

Thanks ! I'll look into it !

There's no way for GKE to manually change the daemonset configuration as it turns out.

Something, someone decided that 50MB should be the memory limit for the pdcsi-node daemonset. Maybe we should bump that limit for GKE 1.31 or next versions. ?

To avoid neediing to recover the journal on the snapshot, you need either to do them offline (ie, after unmounting)

Yeah but for a mission-critical OLTP DB, it's not practical.

quiesce & freeze

Yep. I am using PostgreSQL which is okay-ish with the disk snapshot as they are (without quiesce & freeze). On restore, PG enters a "WAL recovery" step (which can take minutes), but the data is consistent after this.

What bugs me a little is that my "take a disk snapshot and assess PG can recover from that disk snapshot" script used to work fine until GKE v1.28. Something must have changed between v1.27 and v1.28, and now the OOM killer decides more-often-than-not to also kill the gce-pd-driver main process and not only the spawned/child fsck.ext4 process. I couldn't track if that change came from the K8s upgrade, the gcp-compute-persistent-disk-csi-driver upgrade or the COS image upgrade.

mattcary commented 4 months ago

There's no way for GKE to manually change the daemonset configuration as it turns out.

Something, someone decided that 50MB should be the memory limit for the pdcsi-node daemonset. Maybe we should bump that limit for GKE 1.31 or next versions. ?

Most workloads use far less then 50M, increasing the limit makes that memory unusable for user workloads on all nodes on all clusters. We're trying to find a more efficient solution.

Yeah but for a mission-critical OLTP DB, it's not practical.

Yeah, understood.

What bugs me a little is that my "take a disk snapshot and assess PG can recover from that disk snapshot" script used to work fine until GKE v1.28. Something must have changed between v1.27 and v1.28, and now the OOM killer decides more-often-than-not to also kill the gce-pd-driver main process and not only the spawned/child fsck.ext4 process. I couldn't track if that change came from the K8s upgrade, the gcp-compute-persistent-disk-csi-driver upgrade or the COS image upgrade.

Yeah, that's interesting as we'd also only seen the fsck process get oom killed. If it's the kernel oom killer, then I don't think it's due to a change in the kubelet. Maybe it's a cos issue, although the obvious thing, upgrade to cgroup v2, happened in 1.26.

mattcary commented 4 months ago

Do you have the exact 1.27 and 1.28 gke versions you were on (eg v1.27.10-gke.1055000)?

JordanP commented 4 months ago

Do you have the exact 1.27 and 1.28 gke versions you were on (eg v1.27.10-gke.1055000)?

Not exactly. I am now running 1.28.10-gke.1075001 but I don't know which 1.27 version I was running. This nodepool has node autoupgrade enabled. We are on the "static" (a.k.a no) channel. I couldn't find the info a was looking with this Stackdriver search:

protoPayload.methodName="google.container.internal.ClusterManagerInternal.UpdateClusterInternal"
resource.type="gke_nodepool"

The best way to increase the memory for a daemonset instance is to use a mutating webhook

It looks like this is going to work. We have a mutating webhook to increase the container memory limit working in a staging cluster.

FWIW, kernel logs when only fsck.ext4 process is killed:

Kernel logs when both processes are being OOM killed:

JordanP commented 4 months ago

Edit: I finally found it somewhere. We were running 1.27.11-gke.1062004 (cos-105-17412-294-29 according to this mapping)

mattcary commented 4 months ago

Ah, I wonder if it's https://github.com/kubernetes/kubernetes/issues/117070. Let me do some more spelunking. Thank you for the cluster versions you were running, that's very helpful.

On Tue, Jul 16, 2024 at 1:12 PM Jordan Pittier @.***> wrote:

Edit: I finally found it somewhere. We were running 1.27.11-gke.1062004

— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/1782#issuecomment-2231746933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIJCBAETEOBSBH3KSCPYZDLZMV5BFAVCNFSM6AAAAABKS2CVFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRG42DMOJTGM . You are receiving this because you commented.Message ID: <kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/1782/2231746933 @github.com>

mattcary commented 4 months ago

Yes, I think this explains the reason why both processes are killed: https://github.com/kubernetes/kubernetes/pull/117793

The change seems to have happened in 1.28 which explains your problems.

On Tue, Jul 16, 2024 at 1:18 PM Matthew Cary @.***> wrote:

Ah, I wonder if it's https://github.com/kubernetes/kubernetes/issues/117070. Let me do some more spelunking. Thank you for the cluster versions you were running, that's very helpful.

On Tue, Jul 16, 2024 at 1:12 PM Jordan Pittier @.***> wrote:

Edit: I finally found it somewhere. We were running 1.27.11-gke.1062004

— Reply to this email directly, view it on GitHub https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/1782#issuecomment-2231746933, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIJCBAETEOBSBH3KSCPYZDLZMV5BFAVCNFSM6AAAAABKS2CVFCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZRG42DMOJTGM . You are receiving this because you commented.Message ID: <kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/issues/1782/2231746933 @github.com>

JordanP commented 4 months ago

I implemented the workaround described here and it solved my issue.

JordanP commented 4 months ago

Thank you @mattcary, you helped me a lot and you found the root cause :beers:

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

mattcary commented 1 month ago

/lifecycle frozen

azat commented 1 month ago

Maybe this daemonset should not limit memory at all? pdcsi-node can contain not only fsck, but also resize2fs, and if you will kill it, then you should expect all sort of troubles

acondrat commented 1 month ago

We should at least have a way to resize and allocate more memory for it!

mattcary commented 1 month ago

We've found increasing the memory limit to ~500M works well, and hasn't seemed to have an adverse affect on other workloads on the node.

@acondrat the memory resource can be controlled by editing deploy/kubernetes/base/node_linux/node.yaml.

doit-mattporter commented 1 month ago

I've been running into the same issue the past few weeks with the latest stable release GKE version (1.30.5-gke.1014001). I can attach 8 TB hyperdisks to c4 machines just fine, but as soon as I increase the disk size beyond that (e.g. 9 TB) this driver will OOM 100% of the time and get stuck in a loop in what appears to be attempts to run a mkfs ext4 command. I believe I have also seen fsck commands in the logs crashing it as well.

I haven't been able to find a solution beyond avoiding larger disk sizes; this seems like a config fix GCP needs to deploy for GKE, or at least make it straightforward for users to configure.

mattcary commented 1 month ago

See details for GKE mitigation and fix rollout in https://issuetracker.google.com/issues/338125229.

doit-mattporter commented 1 month ago

Upgrading to 1.31.1-gke.1678000 fixed the issue :)

kubernetes-sigs / gcp-compute-persistent-disk-csi-driver

gce-pd-driver container OOM killed after upgrade to GKE 1.28 #1782