Create network resources for AWS Local Zones and Wavelength Zones

[mtulio]

/kind feature

Describe the solution you'd like:

Support of creation network infrastructure, subnets, and gateways, required for AWS Local Zones and Wavelength zones, allowing users to create compute nodes (non-control plane) specifying those subnets.

AWS Local Zones allow end-users to deploy cloud resources closer to the metropolitan regions and offer a limited resource, like Instance Type, Block Storage, Application Load Balancers, Nat Gateways etc. The network infrastructure required to create nodes in Local Zones are:

• public and private subnets,
• public subnet associated with a route table with Internet Gateway as a default router
• private subnet associated with a route table with a valid Nat Gateway*
• *Currently, Nat Gateway is not globally supported by Local Zone locations, and currently there is not API to query if it is offered in the zone.

AWS Wavelength zones are infrastructure running in carrier infrastructure. It also have limitations of resources offered in those locations.

Network resources required for Wavelength zones:

• Carrier Gateway attached to the VPC
• public and private subnets
• public "carrier" route table with the default route entry targeting the Carrier Gateway
• private route table with the default route entry pointing to a valid Nat Gateway*
• public subnet associated with the public "carrier" route table
• private subnet associated with the private route table
• *Same Nat GW limitation as Local Zone.

Not Goal:

• Create control plane nodes, or required resources, in remote zones.
• Compute/Machine resource creation.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Currently, OpenShift creates the network resources required to deploy only compute nodes in AWS Local Zones and Wavelength zones, and we would like to use Cluster API to deploy those resources. Machines (compute nodes) are then created, outside Cluster API.

Considering the limitation of network-based load balancers, we used to recommend to the users to use the AWS Application Load Balancer controller to create ingress traffic into those zones (day-2).

For the gateway for private subnets, we are reusing the same Nat Gateway for the Parent Zone (zone attribute), when it exists, otherwise, the "default" private route table will be used to associate the zone in the remote location. If possible, it would be nice to deploy Nat Gateway in the location, when supported (currently, only a few Local Zone locations support it), otherwise, use some strategy like that.

For the gateway for public subnets in Wavelength Zones, we are creating a single "public carrier" route table, with the default route entry to the Carrier Gateway, previously created and associated with the VPC,

For more details feel free to read the enhancement with more details here.

The suggested Local Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to create the cluster, including Local Zone or Wavelength Zones, in the existing API field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-nyc-1a"}   
  for _, zone := range zones {
  cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
      AvailabilityZone: zone,
      CidrBlock:        privateCIDRs[idxCIDR].String(),
  ID:               fmt.Sprintf("%s-subnet-private-%s", infraID, zone),
  IsPublic:         false,
  })
  cluster.Spec.NetworkSpec.Subnets = append(cluster.Spec.NetworkSpec.Subnets, capa.SubnetSpec{
      AvailabilityZone: zone,
  CidrBlock:        publicCIDRs[idxCIDR].String(),
  ID:               fmt.Sprintf("%s-subnet-public-%s", infraID, zone),
  IsPublic:         true,
  })
  idxCIDR++
  }

• Network reconciliation creates the Carrier Gateway, attaching to the VPC
• Subnet reconciliation creates the public and private subnets into those zones, then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the public subnet "is Wavelength Zone", and find the appropriate route and gateway for public subnets in Carrier Gateways
• Route reconciliation check whether the subnet "is Wavelength Zone", and find the appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the SubnetSpec:

  // ZoneType defines a zone type for this subnet.
  ZoneType *string `json:"zoneType,omitempty"`
  
  // ParentZoneName defines a parent zone name for the zone that the subnet is created,
  // when applied. Available only in zone types local-zone or wavelength-zone.
  ParentZoneName *string `json:"parentZoneName,omitempty"`

The suggested Wavelength Zones workflow looks like this:

• User creates the Subnets (spec.Network.Subnets) specifying all required zones to create the cluster, including Local Zone or Wavelength Zones, in the existing API field SubnetSpec.AvailabilityZone - keeping the valid placeholder ID to teach Cluster API to create subnets.

  zones := []string{"us-east-1a", "us-east-1b", "us-east-1c", "us-east-1-wl1-nyc-wlz-1"}
  ...

• Subnet reconciliation creates the public and private subnets into those zones, then discovers the zone attributes* for the subnet's zone
• Subnets prevent returning subnets created in Wavelength Zone locations to the regular flow, preventing issues in the controllers consuming the standard flow
• Route reconciliation check whether the subnet "is Local Zone", and find the appropriate gateway for private subnets
• Regular cluster creation flow

*Required API changes for the proposed workflow:

• Add the following fields to the VPCSpec:

  // CarrierGatewayID is the id of the Carrier internet gateway associated with the VPC.
  // +optional
  CarrierGatewayID *string `json:"carrierGatewayId,omitempty"`

### Tasks
- [ ] https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4882
- [ ] https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4901
- [ ] 🌱 Implement e2e tests for edge subnets

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten