Are overlapping Subnets for Nodes and Pods possible?

[mganter]

We have to check if overlapping node and pod cidrs can be used.

Example setup 1:

• Two node cluster.
• Node and Pod CIDRs should be equal.
• Nodes should have a delegated prefix (or equivalent)
• Part of the delegated prefix should be the node's pod CIDR
• Part of the delegated prefix should be the node IP (bound to an interface)

Tests: (With Cilium and Calico)

• Can this cluster run properly?
• Can pods reach each other? (Intra- and internode)
• Can pods reach all nodes?

[mganter]

We tested it using Kubeadm on OpenStack VMs:

- Network with 2 subnets
  - 10.0.0.0/8
  - 2001:db8::/110 (without DHCP)

- Nodes
  - 2001:db8::2:1 (master)
  - 2001:db8::3:1 (worker)

- node size cidr ipv6 /112
- Pod cidr 2001:db8::2:0/111
  - 2001:db8::2:0/112 (master)
  - 2001:db8::3:0/112 (worker

We used calico as CNI. As Calico doens't respect the podCidrs field on nodes, we had to manage manually, that the pod cidrs were delegated to the correct nodes. So for calico we had the following setup:

2001:db8::2:1 had 2001:db8::2:1f40/122
2001:db8::3:1 had 2001:db8::3:1f40/122

For calico, remember that you need to have Calico dual stack config in place.

A next test setup forcing the IPAMBlocks to equal 2001:db8::2:0/112 and 2001:db8::3:0/112. This resulted in routing errors!

Not going to work:

• Calico: Configure IPAMBlocks to the same a the delegated prefix (if host ip is inside the delegated prefix)

Unsolved topics:

• We needed to join the workers in the correct order, to have the correct podCidrs assigning to the nodes. So we need to find a solution assigning the podCidrs correctly on node join.
• To use calico in an automated manner, we need to tell calico to respect the podCidrs field on nodes or to attach the correct IPAMBlocks to the nodes.
• To be tested - cilium

[mganter]

New Insights:

The node.spec.podCidr / node.spec.podCidrs assignment from kube-controller-manager can be disable. So that kube-controller-manager does not need the information about the cluster-cidr. Keep in mind, that the network plugin needs to manage the pod cidrs then. (Calico does this anyway, but won't be able to autodetect the IPPool)

Kube-Proxy uses cluster-cidr for service traffic routing, which can be disabled optionally.