Closed fbozic closed 1 year ago
hakman
commented
1 year ago Thanks for the report. I verified and prepare a fix for it in https://github.com/kubernetes/kops/pull/14700. Would help if you could test that change by manually editing the deployment to confirm that there's nothing else missing.
fbozic
commented
1 year ago Hi, thanks for the quick reply.
I've manually tested it by changing CAS deployment. I've opened GCP console, found MIGs that kOps created and translated that to CAS config.
I've noticed that kOps creates 1 MIG per zone even though I have defined 1 kOps IG with n zones. From the source code it looks like kOps doesn't support regional MIGs (yet), hence 1 MIG per zone. Let me know if I'm breaking the design here, and if I should opt out for 1 kOps IG per zone.
Here is an old configuration generate by kOps. There is only one nodes argument which is wrong because my InstanceGroup should be in 3 zones which means kOps creates 3 zonal MIGs.
containers:
- command:
- ./cluster-autoscaler
- --balance-similar-node-groups=false
- --cloud-provider=gce
- --expander=random
- --nodes=3:6:nodes-europe-west3
- --scale-down-utilization-threshold=0.5
- --skip-nodes-with-local-storage=true
- --skip-nodes-with-system-pods=true
- --scale-down-delay-after-add=10m0s
- --scale-down-unneeded-time=10m0s
- --scale-down-unready-time=20m0s
- --new-pod-scale-up-delay=0s
- --max-node-provision-time=15m0s
- --cordon-node-before-terminating=true
- --logtostderr=true
- --stderrthreshold=info
- --v=4Here is my manual configuration. Notice I have 3 nodes arguments.
Since I have define min=3,max=6 on entire kOps IG, each zonal MIG has min=1,max=2.
containers:
- command:
- ./cluster-autoscaler
- --balance-similar-node-groups=false
- --cloud-provider=gce
- --expander=random
- --nodes=1:2:https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-a/instanceGroups/a-nodes-europe-west3-my-fake-name-k8s-local
- --nodes=1:2:https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-b/instanceGroups/b-nodes-europe-west3-my-fake-name-k8s-local
- --nodes=1:2:https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-c/instanceGroups/c-nodes-europe-west3-my-fake-name-k8s-local
- --scale-down-utilization-threshold=0.5
- --skip-nodes-with-local-storage=true
- --skip-nodes-with-system-pods=true
- --scale-down-delay-after-add=10m0s
- --scale-down-unneeded-time=10m0s
- --scale-down-unready-time=20m0s
- --new-pod-scale-up-delay=0s
- --max-node-provision-time=15m0s
- --cordon-node-before-terminating=true
- --logtostderr=true
- --stderrthreshold=info
- --v=4Events
❯ k get events --sort-by='.lastTimestamp'
LAST SEEN TYPE REASON OBJECT MESSAGE
19s Warning FailedScheduling pod/test-fbozic-7c6cf6775c-278vn 0/6 nodes are available: 3 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
19s Warning FailedScheduling pod/test-fbozic-7c6cf6775c-jfsfm 0/6 nodes are available: 3 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
19s Warning FailedScheduling pod/test-fbozic-7c6cf6775c-wr8h8 0/6 nodes are available: 3 node(s) didn't match pod anti-affinity rules, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/6 nodes are available: 3 No preemption victims found for incoming pod, 3 Preemption is not helpful for scheduling.
19s Normal SuccessfulCreate replicaset/test-fbozic-7c6cf6775c Created pod: test-fbozic-7c6cf6775c-jfsfm
19s Normal SuccessfulCreate replicaset/test-fbozic-7c6cf6775c Created pod: test-fbozic-7c6cf6775c-wr8h8
19s Normal SuccessfulCreate replicaset/test-fbozic-7c6cf6775c Created pod: test-fbozic-7c6cf6775c-278vn
19s Normal ScalingReplicaSet deployment/test-fbozic Scaled up replica set test-fbozic-7c6cf6775c to 6 from 3
5s Normal TriggeredScaleUp pod/test-fbozic-7c6cf6775c-278vn pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-a/instanceGroups/a-nodes-europe-west3-my-fake-name-k8s-local 1->2 (max: 2)}]
5s Normal TriggeredScaleUp pod/test-fbozic-7c6cf6775c-jfsfm pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-a/instanceGroups/a-nodes-europe-west3-my-fake-name-k8s-local 1->2 (max: 2)}]
5s Normal TriggeredScaleUp pod/test-fbozic-7c6cf6775c-wr8h8 pod triggered scale-up: [{https://www.googleapis.com/compute/v1/projects/my-project/zones/europe-west3-a/instanceGroups/a-nodes-europe-west3-my-fake-name-k8s-local 1->2 (max: 2)}]Regarding regional/zonal MIGs from the previous comment.
Here is a link to the TODO source code comment about migration to the regional MIGs: https://github.com/kubernetes/kops/blob/29dbd14c74e0168be5707f170babadc94d923c4c/pkg/model/gcemodel/autoscalinggroup.go#L207
Here is terraform resource and it supports distribution across zones: https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_region_instance_group_manager
hakman
commented
1 year ago Regarding 1 MIG per zone, that was more or less related to cluster-autoscaler limitations. To actually work well, it needs to control in which MIG to put the instance, otherwise it's random (based on cloud availability).
/kind bug
I'm trying to set up a new cluster on GCE with cluster-autoscaler enabled. I couldn't find specific cluster-autoscaler GCE docs, so I've tried just to enable it. Let me know if cluster-autoscaler is not supported for GCE and this is not actually a bug.
1. What
kopsversion are you running? The commandkops version, will display this information.Client version: 1.25.3 (git-v1.25.3)2. What Kubernetes version are you running?
kubectl versionwill print the version if a cluster is running or provide the Kubernetes version specified as akopsflag.3. What cloud provider are you using?
GCE
4. What commands did you run? What is the simplest way to reproduce this issue?
5. What happened after the commands executed? Cluster never becomes healthy because cluster-autoscaler pods are in
CrashLoopBackOff.6. What did you expect to happen? Cluster becomes healthy and all pods deployed by kops are running.
7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yamlto display your cluster manifest. You may want to remove your cluster name and other sensitive information.8. Please run the commands with most verbose logging by adding the
-v 10flag. Paste the logs into this report, or in a gist and provide the gist link here. I've only attached info about cluster-autoscaler pods because cluster provisioning without cluster-autoscaler works. Pods are running on master nodes. I've also noticed that pods have env varAWS_REGIONset toeurope-west3. Location is correct but it is not AWS region.Pod status:
Pod logs:
Pod yaml
9. Anything else do we need to know?