I've noticed that GKE node pool autoscaler will evict jobs in progress if autoscaling policy is set to optimize-utilization. Default setting (balanced) doesn't have that problem.
Based on GKE autoscaling policy docs this can be prevented by adding cluster-autoscaler.kubernetes.io/safe-to-evict: "false" annotation.
I've noticed that GKE node pool autoscaler will evict jobs in progress if autoscaling policy is set to
optimize-utilization. Default setting (balanced) doesn't have that problem.Based on GKE autoscaling policy docs this can be prevented by adding
cluster-autoscaler.kubernetes.io/safe-to-evict: "false"annotation.This is not a GKE specific thing: https://kubernetes.io/docs/reference/labels-annotations-taints/#cluster-autoscaler-kubernetes-io-safe-to-evict