kubernetes-sigs / cluster-api-provider-aws

Kubernetes Cluster API Provider AWS provides consistent deployment and day 2 operations of "self-managed" and EKS Kubernetes clusters on AWS.
http://cluster-api-aws.sigs.k8s.io/
Apache License 2.0
646 stars 575 forks source link

Kubernetes nodes cannot be provisioned any more in subnets tagged with sigs.k8s.io/cluster-api-provider-aws/association: secondary #5227

Open cellux opened 5 days ago

cellux commented 5 days ago

/kind bug

What steps did you take and what happened:

Upgraded the CAPA provider to v2.7.1 and then tried to upgrade one of my AWS clusters to a newer Kubernetes version.

During the rolling update of MachineDeployments, CAPA v2.7.1 rejected creation of new EC2 instances saying subnet XXXX belongs to a secondary CIDR block which won't be used to create instances.

What did you expect to happen:

The new EC2 instances are provisioned - as this used to happen before the upgrade to v2.7.1.

Anything else you would like to add:

Downgrading CAPA provider to v2.6.1 resolved the issue.

The problem might be around this code block in pkg/cloud/services/ec2/instance.go:

            tags := converters.TagsToMap(subnet.Tags)
            if tags[infrav1.NameAWSSubnetAssociation] == infrav1.SecondarySubnetTagValue {
                errMessage += fmt.Sprintf(" subnet %q belongs to a secondary CIDR block which won't be used to create instances.", *subnet.SubnetId)
                continue
            }

Environment:

We use four private subnets in AWS which are pre-provisioned by our IT team:

We followed the docs at https://cluster-api-aws.sigs.k8s.io/topics/eks/pod-networking#unmanaged-static-vpc:

We do not want to use the first two subnets for Kubernetes nodes as those are pretty small and could be easily exhausted when we scale out.

k8s-ci-robot commented 5 days ago

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.