`secondaryCidrBlock` allocates nodes in Pods subnets depending on the order of `AWSManagedControlPlane.Spec.Network.Subnets` list

[esierra-stratio]

/kind bug

What steps did you take and what happened:

Set param secondaryCidrBlock and specify AWSManagedControlPlane.Spec.Network.Subnets in object AWSManagedControlPlane alternating subnets dedicated to nodes and pods.

When an instance is created it takes the first private subnet in the list for a specific AZ no matter if it is a secondary subnet (pods) or not (nodes).

What did you expect to happen:

An instance should take the first private subnet in the list for a specific AZ excluding those with the following tag: sigs.k8s.io/cluster-api-provider-aws/association=secondary

Anything else you would like to add:

IMO, a filter excluding the subnet with a specific tag (this case, sigs.k8s.io/cluster-api-provider-aws/association=secondary) is missing during ec2 creation here: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/7eb4851dd9bdeacf039b6609176b81c068891d9a/pkg/cloud/services/ec2/instances.go#L357 Adding something like

subnets = subnets.ExcludeByTag(infrav1.NameAWSSubnetAssociation, infrav1.SecondarySubnetTagValue)

and

// FilterByTag returns a slice containing subnets that not contains specific tag.
func (s Subnets) ExcludeByTag(tagkey string, tagvalue string) (res Subnets) {
    for _, x := range s {
        if val, ok := x.Tags[tagkey]; !(ok && val == tagvalue) {
            res = append(res, x)
        }
    }
    return
}

in networks_types.go seems to work well.

Environment:

• Cluster-api-provider-aws version: v2.0.2
• Kubernetes version: (use kubectl version): v1.24.7
• OS (e.g. from /etc/os-release):

  PRETTY_NAME="Ubuntu 22.04.1 LTS"
  NAME="Ubuntu"
  VERSION_ID="22.04"
  VERSION="22.04.1 LTS (Jammy Jellyfish)"
  VERSION_CODENAME=jammy
  ID=ubuntu
  ID_LIKE=debian
  UBUNTU_CODENAME=jammy

[Skarlso]

/triage accepted

[esierra-stratio]

Any progress here?

[Skarlso]

Not really. We don't have the bandwidth.

[mtougeron]

This is something that I'm still interested in doing but I haven't had a chance to get to yet.

[mtougeron]

FWIW, in case someone else does pick this up, this is a little more complex than originally described. In some cases, like mine, we do want the controlplane to use the secondary subnets (pod). The selection is actually done on the AWS side based on the entire list of subnets assigned to the controlplane. We need to have a way to separate the list of subnets to create vs the list to assign to the controlplane.