search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.88k stars 4.64k forks source link

Switching an existing ASG-based instance group to use `instanceRequirements` rather than `instances` fails #15304

Open danports opened 1 year ago

danports commented 1 year ago

/kind bug

1. What kops version are you running? The command kops version, will display this information. 1.26.2

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag. 1.26.3

3. What cloud provider are you using? AWS

4. What commands did you run? What is the simplest way to reproduce this issue? Deploy a cluster with an instance group that specifies some instance types under spec.mixedInstancesPolicy.instances. Then update the instance group manifest and replace that section with an instanceRequirements section, e.g.:

    instanceRequirements:
      cpu:
        min: "2"
        max: "4"
      memory:
        min: "4G"
        max: "8G"

5. What happened after the commands executed? When running kops update cluster with the updated instance group manifest, it gets stuck in a failure loop:

W0406 18:22:13.945102    2600 autoscalinggroup.go:662] cannot apply changes to AutoScalingGroup: *awstasks.AutoscalingGroup {"Name":null,"Lifecycle":"","Granularity":null,"InstanceProtection":null,"LaunchTemplate":null,"LoadBalancers":null,"MaxInstanceLifetime":null,"MaxSize":null,"Metrics":null,"MinSize":null,"MixedInstanceOverrides":null,"InstanceRequirements":null,"MixedOnDemandAllocationStrategy":"lowest-price","MixedOnDemandBase":null,"MixedOnDemandAboveBase":null,"MixedSpotAllocationStrategy":null,"MixedSpotInstancePools":null,"MixedSpotMaxPrice":null,"Subnets":null,"SuspendProcesses":null,"Tags":null,"TargetGroups":null,"CapacityRebalance":null,"WarmPool":null}
W0406 18:22:14.198944    2600 executor.go:139] error running task "AutoscalingGroup/somegroup" (8m56s remaining to succeed): error updating AutoscalingGroup: ValidationError: The specified OnDemandAllocationStrategy is not valid. The prioritized allocation strategy is not compatible with instance requirements. Valid options are [lowest-price].

FWIW, it doesn't matter whether onDemandAllocationStrategy: lowest-price is included in the manifest - the error is the same.

6. What did you expect to happen? The existing ASG should have been updated with the instance requirements specified.

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

See snippet above.

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

See above.

9. Anything else do we need to know?

Creating a new instance group with instanceRequirements specified works fine. Switching an existing instance group to instanceRequirements also works fine if you delete the existing ASG before running kops update cluster.

vashishkov commented 1 year ago

Can confirm, have same issue

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

danports commented 1 year ago

/remove-lifecycle stale

k8s-triage-robot commented 8 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

srolija commented 6 months ago

We seem to have a similar issue. So just to check, what did you have as MixedOnDemandAllocationStrategy prior to apply? Was it the default prioritized?

In our case we created the group initially with MixedOnDemandAllocationStrategy=prioritized and then wanted to change it to MixedOnDemandAllocationStrategy=lowest-price but it seems like the reconciliation algorithm doesn't check for that.

There is a check for the spot instances here: https://github.com/kubernetes/kops/blob/db26ad5e7f6f65164f731c90c4a1fcea87cd137f/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go#L536-L539

However, there is no such check for the MixedOnDemandAllocationStrategy which seems like the root cause why it is not detected.

Same seems to be the case here: https://github.com/kubernetes/kops/blob/db26ad5e7f6f65164f731c90c4a1fcea87cd137f/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go#L431-L435

And here: https://github.com/kubernetes/kops/blob/db26ad5e7f6f65164f731c90c4a1fcea87cd137f/upup/pkg/fi/cloudup/awstasks/autoscalinggroup.go#L743-L752

srolija commented 6 months ago

/remove-lifecycle rotten

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

srolija commented 2 months ago

/remove-lifecycle rotten