giantswarm / roadmap

Giant Swarm Product Roadmap
https://github.com/orgs/giantswarm/projects/273
Apache License 2.0
3 stars 0 forks source link

Configurable pod CIDR #1481

Closed alex-dabija closed 2 years ago

alex-dabija commented 2 years ago

Story

-As a cluster operator, I want to configure the cluster's pod CIDR in order to ensure that workloads running on the cluster can communicate with systems running in other peered VPCs or onprem networks.

Background

Customers need more control over the IP ranges used by their infrastructure in order to ensure that workloads running in different VPCs or data centers can talk to each other. The source and destination IPs must be part of non-overlapping CIDRs in order to create a TCP connection.

Remarks

This is already implemented for CAPG and the same settings in values.yaml should be used.

Resources

alex-dabija commented 2 years ago

The pod CIDR is configurable. Our current default are the following:

vpcCIDR: 10.0.0.0/16
serviceCIDR: 172.31.0.0/16
podCIDR: 100.64.0.0/12

I created a cluster with the following settings:

vpcCIDR: 10.5.0.0/16
serviceCIDR: 172.16.0.0/16
podCIDR: 172.17.0.0/16

I applied the following manifests:

---
apiVersion: v1
data:
  values: |
    aws: {}
    bastion: {}
    clusterName: alextest04
    controlPlane:
      replicas: 3
    machinePools:
    - instanceType: m5.xlarge
      maxSize: 10
      minSize: 3
      name: machine-pool0
      rootVolumeSizeGB: 300
    network:
      availabilityZoneUsageLimit: 3
      vpcCIDR: 10.5.0.0/16
      serviceCIDR: 172.16.0.0/16
      podCIDR: 172.17.0.0/16
    organization: giantswarm
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: alextest04
  name: alextest04-userconfig
  namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
  name: alextest04
  namespace: org-giantswarm
spec:
  catalog: cluster
  config:
    configMap:
      name: ""
      namespace: ""
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: cluster-aws
  namespace: org-giantswarm
  userConfig:
    configMap:
      name: alextest04-userconfig
      namespace: org-giantswarm
  version: 0.10.0
---
apiVersion: v1
data:
  values: |
    clusterName: alextest04
    organization: giantswarm
kind: ConfigMap
metadata:
  creationTimestamp: null
  labels:
    giantswarm.io/cluster: alextest04
  name: alextest04-default-apps-userconfig
  namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
  labels:
    app-operator.giantswarm.io/version: 0.0.0
    giantswarm.io/cluster: alextest04
  name: alextest04-default-apps
  namespace: org-giantswarm
spec:
  catalog: cluster
  config:
    configMap:
      name: ""
      namespace: ""
    secret:
      name: ""
      namespace: ""
  kubeConfig:
    context:
      name: ""
    inCluster: true
    secret:
      name: ""
      namespace: ""
  name: default-apps-aws
  namespace: org-giantswarm
  userConfig:
    configMap:
      name: alextest04-default-apps-userconfig
      namespace: org-giantswarm
  version: 0.5.5
alex-dabija commented 2 years ago

All the default pods are running:

❯ kubectl get pods -A -o wide  
NAMESPACE     NAME                                                                READY   STATUS      RESTARTS      AGE   IP             NODE                                         NOMINATED NODE   READINESS GATES
giantswarm    chart-operator-6dd5b8d6ff-flzkj                                     1/1     Running     0             29m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   capi-node-labeler-c44fb                                             1/1     Running     0             25m   172.17.2.6     ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   capi-node-labeler-fs4lg                                             1/1     Running     0             25m   172.17.5.47    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   capi-node-labeler-gjs9l                                             1/1     Running     0             25m   172.17.1.177   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   capi-node-labeler-nkb66                                             1/1     Running     0             25m   172.17.4.176   ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   capi-node-labeler-rhkj9                                             1/1     Running     0             25m   172.17.3.109   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   capi-node-labeler-ssbdb                                             1/1     Running     0             25m   172.17.0.212   ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   cert-exporter-daemonset-97jnz                                       1/1     Running     0             25m   172.17.0.80    ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   cert-exporter-daemonset-mp72t                                       1/1     Running     0             25m   172.17.5.6     ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   cert-exporter-daemonset-nx5m5                                       1/1     Running     0             25m   172.17.4.172   ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   cert-exporter-daemonset-nz457                                       1/1     Running     0             25m   172.17.1.215   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-exporter-daemonset-rztxd                                       1/1     Running     0             25m   172.17.3.223   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-exporter-daemonset-s4b2f                                       1/1     Running     0             25m   172.17.2.211   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-exporter-deployment-597dc7c698-dg6tj                           1/1     Running     0             25m   172.17.3.159   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-manager-cainjector-d6dfcd4c4-tc587                             1/1     Running     0             22m   172.17.3.198   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-manager-controller-74fc678f4d-7ckv7                            1/1     Running     0             22m   172.17.3.252   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-manager-webhook-689c687fcb-fzxf8                               1/1     Running     0             22m   172.17.1.234   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   cert-manager-webhook-689c687fcb-qsxp6                               1/1     Running     0             22m   172.17.5.242   ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   cilium-c7fgs                                                        1/1     Running     0             23m   10.5.222.144   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   cilium-gwd2l                                                        1/1     Running     0             23m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   cilium-h7847                                                        1/1     Running     0             23m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   cilium-k6v62                                                        1/1     Running     0             23m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   cilium-operator-7966b46646-6kdqs                                    1/1     Running     0             23m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   cilium-operator-7966b46646-8f9df                                    1/1     Running     0             23m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   cilium-xzs2k                                                        1/1     Running     0             23m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   cilium-zxgfh                                                        1/1     Running     0             23m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   coredns-5b8fd6c6cc-jtz8p                                            1/1     Running     0             34m   172.17.5.145   ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   coredns-5b8fd6c6cc-zpprt                                            1/1     Running     0             34m   172.17.5.78    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   coredns-adopter-jldhn                                               0/1     Completed   0             35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   coredns-controlplane-796cd85558-44dpn                               1/1     Running     0             22m   172.17.4.84    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   coredns-workers-59c47cb8fb-7k6sx                                    1/1     Running     0             22m   172.17.1.171   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   coredns-workers-59c47cb8fb-qhxbk                                    1/1     Running     0             22m   172.17.3.192   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   ebs-csi-controller-5c7f978c79-qqhb5                                 5/5     Running     0             25m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   ebs-csi-node-bdlmp                                                  3/3     Running     0             23m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   ebs-csi-node-fq6hj                                                  3/3     Running     0             23m   10.5.222.144   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   ebs-csi-node-tbsnk                                                  3/3     Running     0             23m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   etcd-ip-10-5-136-8.eu-west-2.compute.internal                       1/1     Running     0             35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   etcd-ip-10-5-201-44.eu-west-2.compute.internal                      1/1     Running     0             33m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   etcd-ip-10-5-68-129.eu-west-2.compute.internal                      1/1     Running     0             31m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   external-dns-67b8c4b9d6-zvpxv                                       2/2     Running     3 (17m ago)   22m   172.17.2.149   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   kiam-agent-2tqpx                                                    1/1     Running     3 (17m ago)   17m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   kiam-agent-5rrhh                                                    1/1     Running     3 (17m ago)   17m   10.5.222.144   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   kiam-agent-x77zl                                                    1/1     Running     3 (17m ago)   17m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   kiam-namespace-annotation-kube-system-69v8q                         0/1     Completed   0             19m   172.17.1.168   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   kiam-server-8mqpb                                                   1/1     Running     1 (17m ago)   17m   172.17.5.26    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   kiam-server-8t26b                                                   1/1     Running     0             17m   172.17.4.205   ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   kiam-server-9stm2                                                   1/1     Running     0             17m   172.17.0.194   ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   kube-apiserver-ip-10-5-136-8.eu-west-2.compute.internal             1/1     Running     0             35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   kube-apiserver-ip-10-5-201-44.eu-west-2.compute.internal            1/1     Running     0             33m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-apiserver-ip-10-5-68-129.eu-west-2.compute.internal            1/1     Running     0             31m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-controller-manager-ip-10-5-136-8.eu-west-2.compute.internal    1/1     Running     1 (33m ago)   35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   kube-controller-manager-ip-10-5-201-44.eu-west-2.compute.internal   1/1     Running     0             33m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-controller-manager-ip-10-5-68-129.eu-west-2.compute.internal   1/1     Running     0             31m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-proxy-6wvm6                                                    1/1     Running     0             35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   kube-proxy-85975                                                    1/1     Running     0             34m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   kube-proxy-czcq4                                                    1/1     Running     0             34m   10.5.222.144   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   kube-proxy-prfc8                                                    1/1     Running     0             34m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   kube-proxy-wwxh8                                                    1/1     Running     0             33m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-proxy-xcgsf                                                    1/1     Running     0             31m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-scheduler-ip-10-5-136-8.eu-west-2.compute.internal             1/1     Running     1 (33m ago)   35m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   kube-scheduler-ip-10-5-201-44.eu-west-2.compute.internal            1/1     Running     0             33m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-scheduler-ip-10-5-68-129.eu-west-2.compute.internal            1/1     Running     0             31m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   kube-state-metrics-b95c9c5d4-d7cr7                                  1/1     Running     0             22m   172.17.2.106   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   metrics-server-6c68b7d978-9rrl8                                     1/1     Running     0             22m   172.17.3.177   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   metrics-server-6c68b7d978-ntx29                                     1/1     Running     0             22m   172.17.1.228   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   net-exporter-6wznx                                                  1/1     Running     0             22m   172.17.2.57    ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   net-exporter-cd2sx                                                  1/1     Running     0             22m   172.17.0.243   ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   net-exporter-nnqm2                                                  1/1     Running     0             22m   172.17.1.77    ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   net-exporter-qpzz8                                                  1/1     Running     0             22m   172.17.5.24    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   net-exporter-rk98g                                                  1/1     Running     0             22m   172.17.4.15    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   net-exporter-sn6v5                                                  1/1     Running     0             22m   172.17.3.244   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   node-exporter-v1-3-1-8nkz8                                          1/1     Running     0             25m   10.5.68.129    ip-10-5-68-129.eu-west-2.compute.internal    <none>           <none>
kube-system   node-exporter-v1-3-1-gcb42                                          1/1     Running     0             25m   10.5.107.172   ip-10-5-107-172.eu-west-2.compute.internal   <none>           <none>
kube-system   node-exporter-v1-3-1-jvjn9                                          1/1     Running     0             25m   10.5.201.44    ip-10-5-201-44.eu-west-2.compute.internal    <none>           <none>
kube-system   node-exporter-v1-3-1-kwfvm                                          1/1     Running     0             25m   10.5.185.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   node-exporter-v1-3-1-nwj8v                                          1/1     Running     0             25m   10.5.136.8     ip-10-5-136-8.eu-west-2.compute.internal     <none>           <none>
kube-system   node-exporter-v1-3-1-vskcm                                          1/1     Running     0             25m   10.5.222.144   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   vertical-pod-autoscaler-admission-controller-7c58b5fcff-49qdh       1/1     Running     0             19m   172.17.1.150   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   vertical-pod-autoscaler-admission-controller-7c58b5fcff-9d4cg       1/1     Running     0             19m   172.17.3.59    ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
kube-system   vertical-pod-autoscaler-recommender-5c5d8db4b9-dzb6d                1/1     Running     0             19m   172.17.1.207   ip-10-5-185-150.eu-west-2.compute.internal   <none>           <none>
kube-system   vertical-pod-autoscaler-updater-744f77fcd6-bwsvn                    1/1     Running     0             19m   172.17.3.219   ip-10-5-222-144.eu-west-2.compute.internal   <none>           <none>
alex-dabija commented 2 years ago

All the pods which have IPs in the nodes CIDR 10.5.0.0/16 are pods with host networking enabled:

❯ kubectl get pods -A -o wide | grep '10\.5\.' | awk '{print $2}' | xargs -n1 kubectl -n kube-system -o json get pod | jq '.spec.hostNetwork'
Error from server (NotFound): pods "chart-operator-6dd5b8d6ff-flzkj" not found
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
❯ kubectl -n giantswarm -o json get pod chart-operator-6dd5b8d6ff-flzkj | jq '.spec.hostNetwork'
true
alex-dabija commented 2 years ago

Nothing else needs to be don