Closed alex-dabija closed 2 years ago
The pod CIDR is configurable. Our current default are the following:
vpcCIDR: 10.0.0.0/16
serviceCIDR: 172.31.0.0/16
podCIDR: 100.64.0.0/12
I created a cluster with the following settings:
vpcCIDR: 10.5.0.0/16
serviceCIDR: 172.16.0.0/16
podCIDR: 172.17.0.0/16
I applied the following manifests:
---
apiVersion: v1
data:
values: |
aws: {}
bastion: {}
clusterName: alextest04
controlPlane:
replicas: 3
machinePools:
- instanceType: m5.xlarge
maxSize: 10
minSize: 3
name: machine-pool0
rootVolumeSizeGB: 300
network:
availabilityZoneUsageLimit: 3
vpcCIDR: 10.5.0.0/16
serviceCIDR: 172.16.0.0/16
podCIDR: 172.17.0.0/16
organization: giantswarm
kind: ConfigMap
metadata:
creationTimestamp: null
labels:
giantswarm.io/cluster: alextest04
name: alextest04-userconfig
namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
labels:
app-operator.giantswarm.io/version: 0.0.0
name: alextest04
namespace: org-giantswarm
spec:
catalog: cluster
config:
configMap:
name: ""
namespace: ""
secret:
name: ""
namespace: ""
kubeConfig:
context:
name: ""
inCluster: true
secret:
name: ""
namespace: ""
name: cluster-aws
namespace: org-giantswarm
userConfig:
configMap:
name: alextest04-userconfig
namespace: org-giantswarm
version: 0.10.0
---
apiVersion: v1
data:
values: |
clusterName: alextest04
organization: giantswarm
kind: ConfigMap
metadata:
creationTimestamp: null
labels:
giantswarm.io/cluster: alextest04
name: alextest04-default-apps-userconfig
namespace: org-giantswarm
---
apiVersion: application.giantswarm.io/v1alpha1
kind: App
metadata:
labels:
app-operator.giantswarm.io/version: 0.0.0
giantswarm.io/cluster: alextest04
name: alextest04-default-apps
namespace: org-giantswarm
spec:
catalog: cluster
config:
configMap:
name: ""
namespace: ""
secret:
name: ""
namespace: ""
kubeConfig:
context:
name: ""
inCluster: true
secret:
name: ""
namespace: ""
name: default-apps-aws
namespace: org-giantswarm
userConfig:
configMap:
name: alextest04-default-apps-userconfig
namespace: org-giantswarm
version: 0.5.5
All the default pods are running:
❯ kubectl get pods -A -o wide
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
giantswarm chart-operator-6dd5b8d6ff-flzkj 1/1 Running 0 29m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-c44fb 1/1 Running 0 25m 172.17.2.6 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-fs4lg 1/1 Running 0 25m 172.17.5.47 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-gjs9l 1/1 Running 0 25m 172.17.1.177 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-nkb66 1/1 Running 0 25m 172.17.4.176 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-rhkj9 1/1 Running 0 25m 172.17.3.109 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system capi-node-labeler-ssbdb 1/1 Running 0 25m 172.17.0.212 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-97jnz 1/1 Running 0 25m 172.17.0.80 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-mp72t 1/1 Running 0 25m 172.17.5.6 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-nx5m5 1/1 Running 0 25m 172.17.4.172 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-nz457 1/1 Running 0 25m 172.17.1.215 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-rztxd 1/1 Running 0 25m 172.17.3.223 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-daemonset-s4b2f 1/1 Running 0 25m 172.17.2.211 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system cert-exporter-deployment-597dc7c698-dg6tj 1/1 Running 0 25m 172.17.3.159 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system cert-manager-cainjector-d6dfcd4c4-tc587 1/1 Running 0 22m 172.17.3.198 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system cert-manager-controller-74fc678f4d-7ckv7 1/1 Running 0 22m 172.17.3.252 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system cert-manager-webhook-689c687fcb-fzxf8 1/1 Running 0 22m 172.17.1.234 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system cert-manager-webhook-689c687fcb-qsxp6 1/1 Running 0 22m 172.17.5.242 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system cilium-c7fgs 1/1 Running 0 23m 10.5.222.144 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system cilium-gwd2l 1/1 Running 0 23m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system cilium-h7847 1/1 Running 0 23m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system cilium-k6v62 1/1 Running 0 23m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system cilium-operator-7966b46646-6kdqs 1/1 Running 0 23m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system cilium-operator-7966b46646-8f9df 1/1 Running 0 23m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system cilium-xzs2k 1/1 Running 0 23m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system cilium-zxgfh 1/1 Running 0 23m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system coredns-5b8fd6c6cc-jtz8p 1/1 Running 0 34m 172.17.5.145 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system coredns-5b8fd6c6cc-zpprt 1/1 Running 0 34m 172.17.5.78 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system coredns-adopter-jldhn 0/1 Completed 0 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system coredns-controlplane-796cd85558-44dpn 1/1 Running 0 22m 172.17.4.84 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system coredns-workers-59c47cb8fb-7k6sx 1/1 Running 0 22m 172.17.1.171 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system coredns-workers-59c47cb8fb-qhxbk 1/1 Running 0 22m 172.17.3.192 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system ebs-csi-controller-5c7f978c79-qqhb5 5/5 Running 0 25m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system ebs-csi-node-bdlmp 3/3 Running 0 23m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system ebs-csi-node-fq6hj 3/3 Running 0 23m 10.5.222.144 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system ebs-csi-node-tbsnk 3/3 Running 0 23m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system etcd-ip-10-5-136-8.eu-west-2.compute.internal 1/1 Running 0 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system etcd-ip-10-5-201-44.eu-west-2.compute.internal 1/1 Running 0 33m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system etcd-ip-10-5-68-129.eu-west-2.compute.internal 1/1 Running 0 31m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system external-dns-67b8c4b9d6-zvpxv 2/2 Running 3 (17m ago) 22m 172.17.2.149 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system kiam-agent-2tqpx 1/1 Running 3 (17m ago) 17m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system kiam-agent-5rrhh 1/1 Running 3 (17m ago) 17m 10.5.222.144 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system kiam-agent-x77zl 1/1 Running 3 (17m ago) 17m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system kiam-namespace-annotation-kube-system-69v8q 0/1 Completed 0 19m 172.17.1.168 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system kiam-server-8mqpb 1/1 Running 1 (17m ago) 17m 172.17.5.26 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system kiam-server-8t26b 1/1 Running 0 17m 172.17.4.205 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system kiam-server-9stm2 1/1 Running 0 17m 172.17.0.194 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system kube-apiserver-ip-10-5-136-8.eu-west-2.compute.internal 1/1 Running 0 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system kube-apiserver-ip-10-5-201-44.eu-west-2.compute.internal 1/1 Running 0 33m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system kube-apiserver-ip-10-5-68-129.eu-west-2.compute.internal 1/1 Running 0 31m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system kube-controller-manager-ip-10-5-136-8.eu-west-2.compute.internal 1/1 Running 1 (33m ago) 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system kube-controller-manager-ip-10-5-201-44.eu-west-2.compute.internal 1/1 Running 0 33m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system kube-controller-manager-ip-10-5-68-129.eu-west-2.compute.internal 1/1 Running 0 31m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-6wvm6 1/1 Running 0 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-85975 1/1 Running 0 34m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-czcq4 1/1 Running 0 34m 10.5.222.144 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-prfc8 1/1 Running 0 34m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-wwxh8 1/1 Running 0 33m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system kube-proxy-xcgsf 1/1 Running 0 31m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system kube-scheduler-ip-10-5-136-8.eu-west-2.compute.internal 1/1 Running 1 (33m ago) 35m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system kube-scheduler-ip-10-5-201-44.eu-west-2.compute.internal 1/1 Running 0 33m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system kube-scheduler-ip-10-5-68-129.eu-west-2.compute.internal 1/1 Running 0 31m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system kube-state-metrics-b95c9c5d4-d7cr7 1/1 Running 0 22m 172.17.2.106 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system metrics-server-6c68b7d978-9rrl8 1/1 Running 0 22m 172.17.3.177 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system metrics-server-6c68b7d978-ntx29 1/1 Running 0 22m 172.17.1.228 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-6wznx 1/1 Running 0 22m 172.17.2.57 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-cd2sx 1/1 Running 0 22m 172.17.0.243 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-nnqm2 1/1 Running 0 22m 172.17.1.77 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-qpzz8 1/1 Running 0 22m 172.17.5.24 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-rk98g 1/1 Running 0 22m 172.17.4.15 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system net-exporter-sn6v5 1/1 Running 0 22m 172.17.3.244 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-8nkz8 1/1 Running 0 25m 10.5.68.129 ip-10-5-68-129.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-gcb42 1/1 Running 0 25m 10.5.107.172 ip-10-5-107-172.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-jvjn9 1/1 Running 0 25m 10.5.201.44 ip-10-5-201-44.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-kwfvm 1/1 Running 0 25m 10.5.185.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-nwj8v 1/1 Running 0 25m 10.5.136.8 ip-10-5-136-8.eu-west-2.compute.internal <none> <none>
kube-system node-exporter-v1-3-1-vskcm 1/1 Running 0 25m 10.5.222.144 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system vertical-pod-autoscaler-admission-controller-7c58b5fcff-49qdh 1/1 Running 0 19m 172.17.1.150 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system vertical-pod-autoscaler-admission-controller-7c58b5fcff-9d4cg 1/1 Running 0 19m 172.17.3.59 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
kube-system vertical-pod-autoscaler-recommender-5c5d8db4b9-dzb6d 1/1 Running 0 19m 172.17.1.207 ip-10-5-185-150.eu-west-2.compute.internal <none> <none>
kube-system vertical-pod-autoscaler-updater-744f77fcd6-bwsvn 1/1 Running 0 19m 172.17.3.219 ip-10-5-222-144.eu-west-2.compute.internal <none> <none>
All the pods which have IPs in the nodes CIDR 10.5.0.0/16
are pods with host networking enabled:
❯ kubectl get pods -A -o wide | grep '10\.5\.' | awk '{print $2}' | xargs -n1 kubectl -n kube-system -o json get pod | jq '.spec.hostNetwork'
Error from server (NotFound): pods "chart-operator-6dd5b8d6ff-flzkj" not found
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
true
❯ kubectl -n giantswarm -o json get pod chart-operator-6dd5b8d6ff-flzkj | jq '.spec.hostNetwork'
true
Nothing else needs to be don
Story
-As a cluster operator, I want to configure the cluster's pod CIDR in order to ensure that workloads running on the cluster can communicate with systems running in other peered VPCs or onprem networks.
Background
Customers need more control over the IP ranges used by their infrastructure in order to ensure that workloads running in different VPCs or data centers can talk to each other. The source and destination IPs must be part of non-overlapping CIDRs in order to create a TCP connection.
Remarks
This is already implemented for CAPG and the same settings in
values.yaml
should be used.Resources