k8s-triage-robot
commented
7 months ago The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
pawcykca
Which component are you using?: cluster-autoscaler
What version of the component are you using?: cluster-autoscaler:v1.27.1
What k8s version are you using (
kubectl version)?:kubectl versionOutput$ kubectl version Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T18:16:20Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.13", GitCommit:"49433308be5b958856b6949df02b716e0a7cf0a3", GitTreeState:"clean", BuildDate:"2023-04-12T12:08:36Z", GoVersion:"go1.19.8", Compiler:"gc", Platform:"linux/amd64"}What environment is this in?: Openstack with Magnum
What did you expect to happen?:
Rapid scaling down of few nodes from the same nodegroup should be performed (few solutions):
*_COMPLETED(to don't cancel/broke previous scale up/down operation) Configurable by the new parameter for a backward compatibilityWhat happened instead?:
When Cluster Autoscaler try to scale down too fast this same Nodegroup - performs scale down operation every 5-15 seconds on Nodes in this nodegroup - then previous scale down operation (Openstack Heat's stack update) is canceled by Openstack Heat. This mainly applies for scaling of
default-workernodegroup which share this same Heat stack asdefault-masternodegroup, because update of this shared stack (scale operation) checks all resources indefault-masternodegroup and then perform update of resources indefault-workernodegroup.How to reproduce it (as minimally and precisely as possible):
apiVersion: apps/v1 kind: Deployment metadata: name: autoscale-app spec: selector: matchLabels: app.kubernetes.io/instance: autoscale-app replicas: 1 template: metadata: labels: app.kubernetes.io/instance: autoscale-app app.kubernetes.io/name: autoscale-app spec: nodeSelector: magnum.openstack.org/nodegroup: "default-worker" containers: - name: autoscale-app image: registry.k8s.io/hpa-example ports: - containerPort: 80 resources: limits: cpu: 1000m requests: cpu: 1000m affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/name operator: In values: - cluster-autoscaler-hpa-test topologyKey: "kubernetes.io/hostname"default-workernodegroup and all Pods inRunningstatusdefault-workernodegroupAnything else we need to know?:
Cluster Autoscaler configuration
- ./cluster-autoscaler - --alsologtostderr - --cloud-provider=magnum - --cluster-name=fb9b32ba-37be-4292-b667-adef4b9c400f - --cloud-config=/config/cloud-config - --node-group-auto-discovery=magnum:role=worker,autoscale - --scan-interval=15s - --scale-down-unneeded-time=5m - --scale-down-delay-after-failure=3m - --scale-down-delay-after-add=5m - --unremovable-node-recheck-timeout=1m - --node-delete-delay-after-taint=15s - --balance-similar-node-groups=true - --emit-per-nodegroup-metrics=true - --balancing-ignore-label=magnum.openstack.org/nodegroup - --balancing-ignore-label=topology.cinder.csi.openstack.org/zone - --balancing-ignore-label=topology.manila.csi.openstack.org/zoneCluster Autoscaler logs
Removing two nodes from this same nodegorup one by one, not in batch (one operation)
Openstack Heat events