Autoscaling not available on GCP

[yilindu27-na]

We used the terraform for GCP to create a Flyte cluster. We found that the node pool on the GKE cluster cannot be scaled down because "pod is blocking scale down because it’s not backed by a controller."

For more details: GCP: Pod is blocking scale down of underutilized node because it doesn’t have a controller, such as a deployment or replicaset. Refer to logs for more details. Recommended actions Set annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "true" for Pod or define a controller, such as a deployment or replicaset, for the Pod

Log: noScaleDown: { ...... reason: { messageId: "no.scale.down.node.pod.kube.system.unmovable"} }

[davidmirror-ops]

I'm curious what Pod(s) in the cluster would not be backed by a controller. The Flyte Taks pods are backed by the flytepropeller controller tied to the flyteworkflow CRD but not sure if that qualifies for GKE's Autoscaler.

According to this it's likely that it refers to a Pod in the kube-system namespace.

Could you share the output of

kubectl get pods -n kube-system -o wide

[yilindu27-na]

I'm not exactly sure which pod to look at. We are looking into that right now. Is there a particular log I can share with you?

In general though we tried to setup autoscaling using the following two methods which neither worked. But when we try this with a separate (non-flyte) GKE cluster at least the first method works. Are either of these methods the right way to setup autoscaling with using this terraform?

Method 1: updating node pool autoscaling params in gke.tf. When we do this we see that the changes to the cluster are applied but autoscaling doesnt actually happen.

node_pools = [
    {
      name         = "default"
      machine_type = "e2-standard-4"
      total_min_count    = 2
      total_max_count    = 4
      # Set to true if you want to enable Image Streaming. Learn more: https://cloud.google.com/kubernetes-engine/docs/how-to/image-streaming to speed up pulling of images
      enable_gcfs  = false 
      autoscaling = true
    }
  ]

In the second method:

cluster_autoscaling = {
    enabled             = true
    autoscaling_profile = "BALANCED"
    max_cpu_cores       = 6
    min_cpu_cores       = 4
    max_memory_gb       = 16
    min_memory_gb       = 8
    gpu_resources       = []
    auto_repair         = true
    auto_upgrade        = true
    disk_size           = 100
    disk_type           = "pd-standard"
    image_type          = "COS_CONTAINERD"    
  }

p.s. we know these two settings are achieving different autoscaling configurations, at this point we just are trying to make it work.

[davidmirror-ops]

Can you check what Pods are scheduled in the node you want CA to remove?

kubectl get pods --all-namespaces --field-selector spec.nodeName=<NODE_NAME>

Then we can figure out what to do depending on how they match the restrictions mentioned in CA's docs

[yilindu27-na]

Thank you for your reply!

We figured out which pods are blocking the autoscaling by looking at the autoscaler log:

Log: noScaleDown: { nodes: [ 0: { node: {4} reason: { messageId: "no.scale.down.node.pod.kube.system.unmovable" parameters: [ 0: "kube-dns-..." ]}} 1: { node: {4} reason: { messageId: "no.scale.down.node.pod.kube.system.unmovable" parameters: [ 0: "l7-default-backend-..." ]}} 2: { node: {4} reason: { messageId: "no.scale.down.node.pod.kube.system.unmovable" parameters: [ 0: "kube-dns-..." ]}}

And, we tried to set PDBs for those pods but its not working.