Intercommunicate pods from 2 different kubernetes clusters

[felipejfc]

lets say I have 2 kubernetes clusters with the following configs:

cluster 1: node cidr: 172.21.0.0/16 non-masquerade-cidr: 100.68.0.0/14

cluster2: node cidr: 172.19.0.0/16 non-masquerade-cidr: 100.64.0.0/14

would it be possible (with calico) for pods in cluster 1 to communicate with pods in cluster 2? I wonder if a route reflector + bgp peers can help in this situation..

The two cluster are located in different aws VPC's connected by a VPN.

regards

[KevDBG]

Hello @gyliu513 ! Thanks for the answer ! I understand the strategy. One more information, how you handle dns resolution between:

Cluster 1 POD A < --- > Cluster 2 POD B

Thanks,

[gyliu513]

@Zophren I think the dns resolution is for services, what do you mean by

Cluster 1 POD A < --- > Cluster 2 POD B

Two pods on different clusters can communicate?

[karstensi]

You can do that in different ways.

If you use coredns in your kubernetes clusters you can look at this:

https://github.com/coredns/kubernetai

On 25/07/18 16:28, Zophren wrote:

Hello @gyliu513 https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fgyliu513&data=02%7C01%7Ckni%40siteimprove.com%7C5de41c16015f45a1451a08d5f23adabf%7Cad30e5bc301d40dba10a0e8d40abe0f9%7C0%7C0%7C636681256949945933&sdata=kzTfRWXQh53ArtBFU3MClvYYaNwsPWYsKST6RELbrAo%3D&reserved=0 ! Thanks for the answer ! I understand the strategy. One more information, how you handle dns resolution between:

Cluster 1 POD A < --- > Cluster 2 POD B

Thanks,

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprojectcalico%2Fcalico%2Fissues%2F908%23issuecomment-407774043&data=02%7C01%7Ckni%40siteimprove.com%7C5de41c16015f45a1451a08d5f23adabf%7Cad30e5bc301d40dba10a0e8d40abe0f9%7C0%7C0%7C636681256949945933&sdata=3cH3JR%2BGPGVFSjbX3JM%2B%2BjoV83eTt9nIsrCELLmBU1k%3D&reserved=0, or mute the thread https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAdnvFpLQKd3nWcI9IUFMORG3C3x0Ax2yks5uKIB8gaJpZM4OVoi1&data=02%7C01%7Ckni%40siteimprove.com%7C5de41c16015f45a1451a08d5f23adabf%7Cad30e5bc301d40dba10a0e8d40abe0f9%7C0%7C0%7C636681256949945933&sdata=R%2BkyKOP74NVacX%2FzjpfJu0m1pHGsCOJhr57h1zJoQmg%3D&reserved=0.

[ffilippopoulos]

@karstensi @gyliu513 I can see how to use kubernetai to discover remote services, but how are you guys handle proxying traffic to those services. I mean that even if you know how to reach pods from another cluster and the remote clusters virtual service ips, don't you still need a way (kube-proxy??) to inject that information inside ipvs or iptables on the host boxes??

[gyliu513]

@ffilippopoulos Federation V2 community now working for federated ingress which can help you case. Please refer to https://github.com/kubernetes-sigs/federation-v2/issues/96 for detail.

[redbaron]

Am I right, that solutions here are all about ensuring connectivity, but leave cross-cluster NetworkPolicy out of the scope?

[gyliu513]

@redbaron Yes, cross-cluster NetworkPolicy is out of the scope here.

Perhaps we can get more comments from @caseydavenport @tmjd for cross-cluster NetworkPolicy

[redbaron]

Unrelated to implementation, what would be the semantic of cross cluster network policies? Network policies gained option to reference namespaces AND pod selector maybe their semantic meaning can be extended to have same meaning regardless of exact cluster, this would require no change to NetworkPolicySpec type . It would work for cases if multiple clusters have same level of work and possibly would be relatively easy to run by running extra calico-node in policy-only mode, but looking into remote cluster APIserver

[nuriel77]

@gyliu513 maybe you can advise. I've added a route to my kube-6 workernode 10.132.0.12 on a node outside of k8s (10.132.0.21):

10.233.87.192/26 via 10.132.0.12 dev eth0

I can see, using traceroute, that packets are indeed routing to 10.132.0.12 workernode when contacting 10.233.87.193.

In addition (having an nginx container running there) I can see packets arriving from 10.132.0.12 to my kube-6 (10.132.0.12) and to the pod at 10.233.87.193. I even see the pod 10.233.87.193 sending packets back to 10.132.0.21, however those never reach on 10.132.0.21 . I've added 10.132.0.21 to ipset group cali40all-hosts, but still no luck:

[root@kube-6 centos]# tcpdump -i eth0 -nn port 80
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
22:06:03.482375 IP 10.132.0.21.41888 > 10.233.87.193.80: Flags [S], seq 2342722493, win 26730, options [mss 8910,sackOK,TS val 25961408 ecr 0,nop,wscale 7], length 0
22:06:03.482467 IP 10.233.87.193.80 > 10.132.0.21.41888: Flags [S.], seq 999588870, ack 2342722494, win 28960, options [mss 1460,sackOK,TS val 8174033 ecr 25954391,nop,wscale 7], length 0
22:06:07.868600 IP 10.233.87.193.80 > 10.132.0.21.41888: Flags [S.], seq 999588870, ack 2342722494, win 28960, options [mss 1460,sackOK,TS val 8178420 ecr 25954391,nop,wscale 7], length 0

Any idea why packets going back from 10.233.87.193 > 10.132.0.21 never arrive?

[gyliu513]

@nuriel77 did you also add a router on 10.132.0.12 to route the network back to 10.132.0.21?

x.x.x.x/26 via 10.132.0.21 dev eth0

[sohailanjum97]

@felipejfc that should essentially work - depends a bit on what your requirements are. Like @ozdanborne, there are a few ways to go about this.

You could, for example, run a single Calico network across the two clusters and things should "just work" - this has its own caveats w.r.t splitting the control plane across multiple geographic locations, if your VPCs aren't in the same region.

For two separate clusters, you could probably set up BGP peering across the two VPCs so that Calico instances in VPC1 know about pod IPs in VPC2 (haven't tried this myself though)

Like @ozdanborne, the tricky bit is getting policy to work in that scenario.

You did this afterwards ??

[sohailanjum97]

Hello Everyone, I have to make a Bridge Node between Two different kubernetes Clusters. I need some suggestion ... If any one alrady did that..