We need the ability to specify networking details for each VSphereMachineConfig within a single EKS Anywhere cluster

[nickperry]

What would you like to be added: Currently EKS Anywhere vsphere provider assumes all nodes will exist in a single network, defined in spec.network of VSphereDatacenterConfig. It also assumes DHCP will be used.

We need the ability to specify networking details for each VSphereMachineConfig within a single EKS Anywhere cluster, such that we can disable DHCP, specify nameservers and gateways and set the vsphere network, etc.

We would like this to be achieved via an optional network field in the spec of VSphereMachineConfig.

This network struct should map to the resultant VSphereMachineTemplates' spec.template.spec.network.

Why is this needed: We are currently a large VMWare Tanzu customer and are interested in using EKS Anywhere (we are also a large AWS customer). However, the lack of flexibility in network configuration means that we cannot use EKS Aywhere currently. Our clusters have multiple classes of nodes. One of these node classes is an Istio gateway node, which must be deployed in a separate "DMZ" vsphere network. Additionally, none of our networks use DHCP - we disable it on our VSphereMachineTemplates and use an Infoblox IPAM controller, running on the management cluster instead.

We can do this with upstream CAPI and Tanzu via a separate VSphereMachineTemplate directly, however, AWS have made a design decision with EKS Anywhere to implement custom EKS Anywhere CRDs, which do not expose the full flexibility of the underlying CAPI / CAPV APIs.

An example of the network spec within our VSphereMachineTemplates:

      network:
        devices:
        - dhcp4: false
          gateway4: #@ "{}".format(data.values.DEFAULT_GATEWAY_DMZ)
          nameservers:
          - #@ "{}".format(data.values.DNS_RESOLVER_1)
          - #@ "{}".format(data.values.DNS_RESOLVER_2)
          - #@ "{}".format(data.values.DNS_RESOLVER_3)
          networkName: #@ "{}".format(data.values.VSPHERE_NETWORK_DMZ)
          searchDomains:
          - #@ "{}".format(data.values.INFOBLOX_DNS_ZONE)

[nickperry]

If AWS do not want to add network configuration to apiVersion: anywhere.eks.amazonaws.com kind: VSphereMachineConfig, could we still get support from AWS for a cluster where some of the nodes are configured directly using the underlying VSphereMachineTemplate rather than EKS Anywhere's abstraction?

[nickperry]

The simplified abstractions of anywhere.eks.amazonaws.com/v1alpha1 VSphereMachineConfig and VSphereDatacenterConfig preclude the use of https://github.com/telekom/cluster-api-ipam-provider-in-cluster.

If we were working directly with infrastructure.cluster.x-k8s.io/v1beta1 VSphereMachineTemplate resources, we could just set

spec:
  template:
    spec:
      network:
        devices:
        - dhcp4: false
          networkName: foo
          addressesFromPools:
            - apiGroup: ipam.cluster.x-k8s.io
              kind: InClusterIPPool
              name: foo