CAPA granular network - Githubissues

alex-dabija commented 1 year ago

Story

-As a cluster admin, I want to create a CAPA cluster with a granular network layout (separate subnets for the control plane nodes, worker nodes, Kubernetes API loadbalancer, ingress controller, bastion hosts), in order improve the security by configuring CIDR & subnet purpose based firewall rules on the transit gateway.

Background

Some customers want their inter-VPC network traffic to be routed by a transit gateway in order to have control over which systems talk to each other.

Tasks

freudl commented 1 year ago

It would be great to understand how existing assets are placed once there is more than one subnet per AZ.

Also, it would help to understand how we will exchange configuration of VPC internals, e.g. CIDRs and Tags.

alex-dabija commented 1 year ago

It would be great to understand how existing assets are placed once there is more than one subnet per AZ.

We are currently investigating the implications of putting the machine pools and the control plane nodes on separate subnets. I'll let you know as soon as I have more information.

For the moment, I know that as soon as we have multiple subnets on per AZ we'll have to decide where the different types of machines / loadbalancers / VPC endpoints go.

Also, it would help to understand how we will exchange configuration of VPC internals, e.g. CIDRs and Tags.

We are evaluating a few options and we have to draft PRs:

https://github.com/giantswarm/cluster-aws/pull/195/files (look for helm/cluster-aws/values.yaml);
https://github.com/giantswarm/cluster-aws/pull/196/files (look for helm/cluster-aws/values.yaml)).

The basic idea is to have a list of subnets with CIDRs and tags. The different sub-systems (control plane, machine pools, ingress LB, etc.) will use tag filters to lookup the subnets to use.

alex-dabija commented 1 year ago

We have an initial implementation that's able to create and configure a private cluster to use separate subnets for the API LB, the control plane nodes, the worker nodes, the VPC endpoints and the bastion hosts.

Here's an example of a cluster definition:

---
apiVersion: v1
data:
  values: |
    clusterName: alextest43
    organization: giantswarm
    clusterDescription: "Alex's test cluster 40"
    aws:
      region: eu-north-1
    proxy:
      enabled: true
      http_proxy: http://internal-a958df4705a504070a6d173afb0f2258-1466082418.eu-north-1.elb.amazonaws.com:4000
      https_proxy: http://internal-a958df4705a504070a6d173afb0f2258-1466082418.eu-north-1.elb.amazonaws.com:4000
    bastion:
      subnetTags:
      - subnet.giantswarm.io/role: bastion
    controlPlane:
      replicas: 3
      subnetTags:
      - subnet.giantswarm.io/role: control-plane
    machinePools:
    - instanceType: m5.xlarge
      maxSize: 10
      minSize: 3
      name: machine-pool0
      rootVolumeSizeGB: 300
      subnetTags:
      - subnet.giantswarm.io/role: workers
    network:
      availabilityZoneUsageLimit: 3
      vpcCIDR: 10.234.0.0/16
      vpcMode: private
      apiMode: private
      dnsMode: private
      topologyMode: GiantSwarmManaged
      subnets:
      # api-server subnets [HACK: must come first in the list]
      - cidrBlocks:
        - cidr: 10.234.0.0/24
          availabilityZone: a
        - cidr: 10.234.1.0/24
          availabilityZone: b
        - cidr: 10.234.2.0/24
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: api-server-elb

      # Control plane nodes subnets
      - cidrBlocks:
        - cidr: 10.234.4.0/22
          availabilityZone: a
        - cidr: 10.234.8.0/22
          availabilityZone: b
        - cidr: 10.234.12.0/22
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: control-plane

      # Bastion nodes subnets
      - cidrBlocks:
        - cidr: 10.234.16.0/24
          availabilityZone: a
        - cidr: 10.234.17.0/24
          availabilityZone: b
        - cidr: 10.234.18.0/24
          availabilityZone: c
        isPublic: true
        tags:
          subnet.giantswarm.io/role: bastion

      # vpc endpoints and (eventually) transit gateway 
      - cidrBlocks:
        - cidr: 10.234.19.0/24
          availabilityZone: a
        - cidr: 10.234.20.0/24
          availabilityZone: b
        - cidr: 10.234.21.0/24
          availabilityZone: c
        isPublic: true
        tags:
          subnet.giantswarm.io/role: attatchments
          subnet.giantswarm.io/endpoints: "true"

      # Worker nodes subnets
      - cidrBlocks:
        - cidr: 10.234.32.0/19
          availabilityZone: a
        - cidr: 10.234.64.0/19
          availabilityZone: b
        - cidr: 10.234.96.0/19
          availabilityZone: c
        isPublic: false
        tags:
          subnet.giantswarm.io/role: workers
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: alextest43
  name: alextest43-userconfig
  namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
  name: alextest43
  namespace: org-giantswarm
spec:
  catalog: cluster-test
  config:
    configMap:
      name: ""
      namespace: ""
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: cluster-aws
  namespace: org-giantswarm
  userConfig:
    configMap:
      name: alextest43-userconfig
      namespace: org-giantswarm
  version: 0.20.7-3e9a0d09f8e420faad1e6e7c45ca33419f8509ac
---
apiVersion: v1
data:
  values: |
    clusterName: alextest43
    organization: giantswarm
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: alextest43
  name: alextest43-default-apps-userconfig
  namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
    giantswarm.io/cluster: alextest43
    giantswarm.io/managed-by: cluster
  name: alextest43-default-apps
  namespace: org-giantswarm
spec:
  catalog: cluster
  config:
    configMap:
      name: alextest43-cluster-values
      namespace: org-giantswarm
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: default-apps-aws
  namespace: org-giantswarm
  userConfig:
    configMap:
      name: alextest43-default-apps-userconfig
      namespace: org-giantswarm
  version: 0.14.0

alex-dabija commented 1 year ago

There are a few aspects left to take care of:

the API LB subnets need to be the first ones in the list because CAPA just picks the first subnet for a given AZ;
CAPA overwrites the subnet names, which means that it doesn't let us set custom names. This is prevents us from starting the ingress on a specific subnet because all the subnets have the same name for a given AZ;
the trasit gateway operarator needs to be updated in order to choose the correct subnet (currently, in progress);
the implementation still needs to be reviewed;
we still need to ensure backwards compatibility with existing clusters.

alex-dabija commented 1 year ago

Tested with thor when it was recreated last week.

giantswarm / roadmap

CAPA granular network #1782

Story

Background

Tasks