Pods communication not working when running locally

Hi folks 👋

Thanks for the great work you've been doing with Submariner.

I've been playing around with it over that past days and am actually facing an issue with pods communication. I probably missed something.

I created 3 clusters, 2 kind clusters deployed locally onto two different machines, and 1 remotely which stands as the broker. Let's call them client-0, client-1 and broker. Both client-0 and client-1 have 1 control-plane et 1 worker node.

Everything gets deployed properly and the logs seem to state that everything works well. The worker nodes get also labelled properly.

Then I created and exposed an nginx deployment in namespace nginx inside of cluster client-1, that I exported with subctl. So far so good, the ServiceImport is properly being created in client-0. It's looks like this:

apiVersion: v1
items:
- apiVersion: lighthouse.submariner.io/v2alpha1
  kind: ServiceImport
  metadata:
    annotations:
      origin-name: nginx
      origin-namespace: nginx
    creationTimestamp: "2020-08-18T09:03:16Z"
    generation: 1
    managedFields:
    - apiVersion: lighthouse.submariner.io/v2alpha1
      fieldsType: FieldsV1
      fieldsV1:
        f:metadata:
          f:annotations:
            .: {}
            f:origin-name: {}
            f:origin-namespace: {}
        f:spec:
          .: {}
          f:ports: {}
          f:sessionAffinity: {}
          f:sessionAffinityConfig: {}
          f:type: {}
        f:status:
          .: {}
          f:clusters: {}
      manager: lighthouse-agent
      operation: Update
      time: "2020-08-18T09:03:16Z"
    name: nginx-nginx-client-1
    namespace: submariner-operator
    resourceVersion: "960"
    selfLink: /apis/lighthouse.submariner.io/v2alpha1/namespaces/submariner-operator/serviceimports/nginx-nginx-client-1
    uid: aa5824aa-1ceb-44bf-9ed5-4441c17dd7d4
  spec:
    ports: null
    sessionAffinity: ""
    sessionAffinityConfig: null
    type: SuperclusterIP
  status:
    clusters:
    - cluster: client-1
      ips:
      - 10.1.251.197
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

If I now create a random pod inside of client-0 and run getent hosts nginx.nginx.svc.supercluster.local which gives me the following output:

$ getent hosts nginx.nginx.svc.supercluster.local
10.1.251.197    nginx.nginx.svc.supercluster.local

So service discovery seems to be operating properly as well. However, if I try to curl the service, I get the following error:

$ curl nginx.nginx.svc.supercluster.local
curl: (7) Failed to connect to nginx.nginx.svc.supercluster.local port 80: No route to host

I'm not 100% sure but should my local machines have a publicly reachable IP to make StrongSwan work ? If yes, do you know how I could explicitly set it up when joining the broker ?

Otherwise do you have any hints about the reason why communication does not work properly ?

Thanks in advance

I've been playing around with it over that past days and am actually facing an issue with pods communication. I probably missed something.

Thanks @emaincourt for trying out Submariner.

So service discovery seems to be operating properly as well. However, if I try to curl the service, I get the following error:
$ curl nginx.nginx.svc.supercluster.local
curl: (7) Failed to connect to nginx.nginx.svc.supercluster.local port 80: No route to host

Can you please check the output of the following in client-0 or client-1 clusters which will tell us if the IPsec tunnels are properly established. kubectl describe Gateway -n submariner-operator

I'm not 100% sure but should my local machines have a publicly reachable IP to make StrongSwan work ? If yes, do you know how I could explicitly set it up when joining the broker ?

I'm afraid, it's not required. Submariner supports connectivity between two OnPremise clusters (i.e., client-0 and client-1 in your case) While joining the clusters, did you specify "--disable-nat"? If not, this could be the reason why connectivity is broken.

subctl join ... --disable-nat For more details, please take a look at https://submariner.io/quickstart/kind/#locally-testing-with-kind

@sridhargaddam Thanks a lot for your answering so quickly.

Please find below the output of kubectl describe Gateway -n submariner-operator for client-0:

Name:         client-0-worker
Namespace:    submariner-operator
Labels:       <none>
Annotations:  update-timestamp: 1597755012
API Version:  submariner.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2020-08-18T12:25:55Z
  Generation:          2
  Managed Fields:
    API Version:  submariner.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:update-timestamp:
    Manager:         submariner-engine
    Operation:       Update
    Time:            2020-08-18T12:50:12Z
  Resource Version:  17875
  Self Link:         /apis/submariner.io/v1/namespaces/submariner-operator/gateways/client-0-worker
  UID:               7a92887e-30b8-4bb1-aeb7-9c317f377456
Status:
  Connections:
    Endpoint:
      Backend:      strongswan
      cable_name:   submariner-cable-client-1-172-18-0-2
      cluster_id:   client-1
      Hostname:     client-1-worker
      nat_enabled:  true
      private_ip:   172.18.0.2
      public_ip:    89.3.199.26
      Subnets:
        10.1.0.0/16
        10.0.0.0/16
    Status:          connecting
    Status Message:  Connecting to 89.3.199.26:500
  Ha Status:         active
  Local Endpoint:
    Backend:      strongswan
    cable_name:   submariner-cable-client-0-172-18-0-2
    cluster_id:   client-0
    Hostname:     client-0-worker
    nat_enabled:  true
    private_ip:   172.18.0.2
    public_ip:    89.3.199.26
    Subnets:
      10.43.0.0/16
      10.42.0.0/16
  Status Failure:
  Version:         v0.5.0
Events:            <none>

You were right, it definitely looks like the client can't connect to remote host. I assume this connectivity issue comes from the broker not being reachable ?

I recreated both the clients with --disable-nat and the error seems to be of different kind, which is probably good news. Please find below the description of the Gateway after recreating client-0:

Name:         client-0-worker
Namespace:    submariner-operator
Labels:       <none>
Annotations:  update-timestamp: 1597755570
API Version:  submariner.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2020-08-18T12:57:42Z
  Generation:          4
  Managed Fields:
    API Version:  submariner.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:update-timestamp:
    Manager:         submariner-engine
    Operation:       Update
    Time:            2020-08-18T12:59:30Z
  Resource Version:  1262
  Self Link:         /apis/submariner.io/v1/namespaces/submariner-operator/gateways/client-0-worker
  UID:               0456c5ac-c660-449e-aefc-3af787e975db
Status:
  Connections:
    Endpoint:
      Backend:      strongswan
      cable_name:   submariner-cable-client-1-172-18-0-2
      cluster_id:   client-1
      Hostname:     client-1-worker
      nat_enabled:  false
      private_ip:   172.18.0.2
      public_ip:
      Subnets:
        10.1.0.0/16
        10.0.0.0/16
    Status:          error
    Status Message:  No IKE SA found for cable submariner-cable-client-1-172-18-0-2
  Ha Status:         active
  Local Endpoint:
    Backend:      strongswan
    cable_name:   submariner-cable-client-0-172-18-0-2
    cluster_id:   client-0
    Hostname:     client-0-worker
    nat_enabled:  false
    private_ip:   172.18.0.2
    public_ip:
    Subnets:
      10.43.0.0/16
      10.42.0.0/16
  Status Failure:
  Version:         v0.5.0
Events:            <none>

I found out a related issue. However, event after waiting for several minutes, the error does not disappear whereas the logs from the gateway pod look good:

I0818 13:04:43.734092       1 cableengine.go:127] Successfully installed Endpoint cable "submariner-cable-client-1-172-18-0-2" with remote IP 172.18.0.2
I0818 13:04:43.734274       1 tunnel.go:106] Tunnel controller successfully installed Endpoint cable submariner-cable-client-1-172-18-0-2 in the engine
I0818 13:04:43.742820       1 kubernetes.go:340] Cluster "client-0" matched what we received from datastore - not updating
I0818 13:04:43.792214       1 kubernetes.go:392] Endpoint "client-0-submariner-cable-client-0-172-18-0-2" matched what we received from datastore - not updating
I0818 13:04:43.802589       1 datastoresyncer.go:260] The updated submariner Endpoint "client-1" is not for this cluster - skipping updating the datastore
I0818 13:04:44.229534       1 kubernetes.go:260] UpdateFunc in WatchEndpoints called
I0818 13:04:44.229735       1 datastoresyncer.go:370] In reconcileEndpointCRD: &types.SubmarinerEndpoint{Spec:v1.EndpointSpec{ClusterID:"client-1", CableName:"submariner-cable-client-1-172-18-0-2", Hostname:"client-1-worker", Subnets:[]string{"10.1.0.0/16", "10.0.0.0/16"}, PrivateIP:"172.18.0.2", PublicIP:"", NATEnabled:false, Backend:"strongswan", BackendConfig:map[string]string(nil)}}
I0818 13:04:44.230454       1 kubernetes.go:177] UpdateFunc in WatchClusters called
I0818 13:04:44.230486       1 kubernetes.go:177] UpdateFunc in WatchClusters called
I0818 13:04:44.230512       1 datastoresyncer.go:297] In reconcileClusterCRD: &types.SubmarinerCluster{ID:"client-1", Spec:v1.ClusterSpec{ClusterID:"client-1", ColorCodes:[]string{"blue"}, ServiceCIDR:[]string{"10.1.0.0/16"}, ClusterCIDR:[]string{"10.0.0.0/16"}, GlobalCIDR:[]string{}}}
I0818 13:04:44.241394       1 datastoresyncer.go:417] Endpoint "client-1-submariner-cable-client-1-172-18-0-2" matched what we received from datastore - not updating
I0818 13:04:44.242093       1 kubernetes.go:260] UpdateFunc in WatchEndpoints called
I0818 13:04:44.250014       1 datastoresyncer.go:344] Cluster "client-1" matched what we received from datastore - not updating

Do you have any idea what's happening ?

Thanks in advance

@sridhargaddam Thanks a lot for your answering so quickly.

No problem, most welcome.

You were right, it definitely looks like the client can't connect to remote host. I assume this connectivity issue comes from the broker not being reachable?

Submariner requires that Broker cluster be accessible from all the member clusters (aka client-0 and client-1). From the logs you shared, I don't think there is an issue with accessing the Broker cluster.

I recreated both the clients with --disable-nat and it seems the error seems to be of different kind, which is probably good news. Please find below the description of the Gateway after recreating client-0:

This is where I see something weird. The private_ip: of both client-1 and client-0 clusters is showing the same ip-address 172.18.0.2. The private_ip: reported in the Gateway object is nothing but the local interface address of your Gateway node in each of the clusters. The IPsec tunnel is not getting established because the ip-address to which we are trying to establish the connection is the same. Please check your setup on why this is happening.

JFYI, Submariner code retrieves the local network address in the following manner https://github.com/submariner-io/submariner/blob/8ad86f5c25aff4646940f945edd161171b04c519/pkg/util/util.go#L60

In a working KIND setup with two clusters (cluster1 and cluster2), the Gateway object will be as shown.

[sgaddam@localhost submariner-operator]$ kubectl describe Gateway -n submariner-operator
Name:         cluster1-worker
Namespace:    submariner-operator
Labels:       <none>
Annotations:  update-timestamp: 1597757895
API Version:  submariner.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2020-08-18T13:33:39Z
  Generation:          56
  Resource Version:    2361
  Self Link:           /apis/submariner.io/v1/namespaces/submariner-operator/gateways/cluster1-worker
  UID:                 96ea8270-c6b3-4d68-a8b2-fb8a3e3689d3
Status:
  Connections:
    Endpoint:
      Backend:      strongswan
      Cable Name:   submariner-cable-cluster2-172-17-0-4
      Cluster Id:   cluster2
      Hostname:     cluster2-worker
      Nat Enabled:  false
      Private Ip:   172.17.0.4
      Public Ip:    
      Subnets:
        100.92.0.0/16
        10.242.0.0/16
    Status:          connected
    Status Message:  Connected to 172.17.0.4:4500 - encryption alg=AES_GCM_16, keysize=128 rekey-time=14082
  Ha Status:         active
  Local Endpoint:
    Backend:      strongswan
    Cable Name:   submariner-cable-cluster1-172-17-0-7
    Cluster Id:   cluster1
    Hostname:     cluster1-worker
    Nat Enabled:  false
    Private Ip:   172.17.0.7
    Public Ip:    
    Subnets:
      100.91.0.0/16
      10.241.0.0/16
  Status Failure:  
  Version:         v0.5.0-30-g8ad86f5-dev
Events:            <none>
[sgaddam@localhost submariner-operator]$

[sgaddam@localhost submariner-operator]$ kubectl get nodes -o wide 
NAME                     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE       KERNEL-VERSION           CONTAINER-RUNTIME
cluster1-control-plane   Ready    master   68m   v1.17.0   172.17.0.9    <none>        Ubuntu 19.10   5.7.14-200.fc32.x86_64   containerd://1.3.2
cluster1-worker          Ready    <none>   67m   v1.17.0   172.17.0.7    <none>        Ubuntu 19.10   5.7.14-200.fc32.x86_64   containerd://1.3.2
cluster1-worker2         Ready    <none>   67m   v1.17.0   172.17.0.5    <none>        Ubuntu 19.10   5.7.14-200.fc32.x86_64   containerd://1.3.2

@sridhargaddam Thanks a lot. You're actually right, both the clusters run on a Docker network with the same CIDR.

I've delete all the 3 clusters. Then I've created a new Docker network with CIDR 172.19.0.0/16 for client-0 on machine 0, left the network with CIDR 172.17.0.0/16 untouched for client-1 on machine 1, and a new remote broker cluster for which I opened all incoming and outcoming TCP/UDP connections.

I've ran the following commands then:

# broker
subctl deploy-broker --service-discovery
# client-0
subctl join broker-info.subm --clusterid client-0 --disable-nat
# client-1
subctl join broker-info.subm --clusterid client-1 --disable-nat

Now, if I describe my Gateway on client-0, I still get:

Name:         client-0-worker
Namespace:    submariner-operator
Labels:       <none>
Annotations:  update-timestamp: 1597761505
API Version:  submariner.io/v1
Kind:         Gateway
Metadata:
  Creation Timestamp:  2020-08-18T14:32:43Z
  Generation:          4
  Managed Fields:
    API Version:  submariner.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          f:update-timestamp:
    Manager:         submariner-engine
    Operation:       Update
    Time:            2020-08-18T14:38:25Z
  Resource Version:  3478
  Self Link:         /apis/submariner.io/v1/namespaces/submariner-operator/gateways/client-0-worker
  UID:               d549193c-e82c-485f-80a8-517339592ded
Status:
  Connections:
    Endpoint:
      Backend:      strongswan
      cable_name:   submariner-cable-client-1-172-18-0-3
      cluster_id:   client-1
      Hostname:     client-1-worker
      nat_enabled:  false
      private_ip:   172.18.0.3
      public_ip:
      Subnets:
        10.1.0.0/16
        10.0.0.0/16
    Status:          connecting
    Status Message:  Connecting to 172.18.0.3:500
  Ha Status:         active
  Local Endpoint:
    Backend:      strongswan
    cable_name:   submariner-cable-client-0-172-19-0-3
    cluster_id:   client-0
    Hostname:     client-0-worker
    nat_enabled:  false
    private_ip:   172.19.0.3
    public_ip:
    Subnets:
      10.43.0.0/16
      10.42.0.0/16
  Status Failure:
  Version:         v0.5.0
Events:            <none>

Same for client-1. Logs of the gateway pod look not so bad except for the following line if it might help:

W0818 14:32:44.314500 1 strongswan.go:531] Failed to connect to charon - retrying: dial unix /var/run/charon.vici: connect: no such file or directory

I'm definitely not sure if it matters. Despite this log, StrongSwan still seems to operate. If I connect to the gateway pod on client-0 and run strongswan status-all, I get the following output:

$ strongswan statusall
Status of IKE charon daemon (strongSwan 5.8.4, Linux 4.19.76-linuxkit, x86_64):
  uptime: 23 minutes, since Aug 18 14:32:43 2020
  malloc: sbrk 3002368, mmap 0, used 1140352, free 1862016
  worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 1
  loaded plugins: charon pkcs11 tpm aesni aes des rc2 sha2 sha1 md4 md5 mgf1 random nonce x509 revocation constraints acert pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl gcrypt fips-prf gmp curve25519 chapoly xcbc cmac hmac ctr ccm gcm drbg newhope curl attr kernel-netlink resolve socket-default farp stroke vici updown eap-identity eap-sim eap-aka eap-aka-3gpp eap-aka-3gpp2 eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap xauth-generic xauth-eap xauth-pam xauth-noauth dhcp led duplicheck unity counters
Listening IP addresses:
  10.42.1.1
  10.42.1.1
  10.42.1.1
  10.42.1.1
  172.19.0.3
  240.19.0.3
Connections:
submariner-cable-client-1-172-18-0-3:  172.19.0.3...172.18.0.3  IKEv2
submariner-cable-client-1-172-18-0-3:   local:  [172.19.0.3] uses pre-shared key authentication
submariner-cable-client-1-172-18-0-3:   remote: [172.18.0.3] uses pre-shared key authentication
submariner-child-submariner-cable-client-1-172-18-0-3:   child:  172.19.0.3/32 10.43.0.0/16 10.42.0.0/16 === 172.18.0.3/32 10.1.0.0/16 10.0.0.0/16 TUNNEL
Security Associations (0 up, 1 connecting):
submariner-cable-client-1-172-18-0-3[2]: CONNECTING, 172.19.0.3[%any]...172.18.0.3[%any]
submariner-cable-client-1-172-18-0-3[2]: IKEv2 SPIs: 0f6007d8e3ec35e3_i* 0000000000000000_r
submariner-cable-client-1-172-18-0-3[2]: Tasks active: IKE_VENDOR IKE_INIT IKE_NATD IKE_CERT_PRE IKE_AUTH IKE_CERT_POST IKE_CONFIG CHILD_CREATE IKE_AUTH_LIFETIME

My StrongSwan knowledge is pretty small but I don't really understand how it is supposed to get connected to 172.18.0.3 as this IP is not publicly reachable and both the hosts are not located on the same machine. What is the trick here ?

However this time the Gateway looks pretty close to what you got so I guess we might be close to working.

Any thoughts ?

Thanks once again for your time

@sridhargaddam Thanks a lot. You're actually right, both the clusters run on a Docker network with the same CIDR.

Actually, even if you have all your clusters on the same Docker network, it will work (though it's not a realistic setup). This is how submariner KIND setup is deployed for developer testing (i.e., JFYI in submariner repo, you can simply run "make deploy" and it will create three clusters and deploy submariner). Anyways, if you created a different docker network it's fine.

A sample KIND deployment with Submariner looks like this. Screenshot from 2020-08-18 20-28-01

@sridhargaddam Thanks for the diagram ! So in your case all the clusters run on the same network. Then basically it makes sense Cluster2 and Cluster3 can communicate with each other through StrongSwan. In my case, they run on private networks, on different hosts and can, I think, therefore not communicate with each other. Am I right and/or clear ?

My StrongSwan knowledge is pretty small but I don't really understand how it is supposed to get connected to 172.18.0.3 as this IP is not publicly reachable and both the hosts are not located on the same machine. What is the trick here ?

mmm... When you said, KIND clusters, I was assuming that your KIND clusters were on the same machine. If the clusters are on different machines, then you are right, client-0 cannot connect to the private-ip 172.18.0.3 as the IP is not reachable from outside of the machine.

Based on what you explained, I can describe your setup as follows. Two clusters onPrem but they don't have direct ip-reachability and are behind the same NAT-Router (to internet).

@mangelajo any idea if this use-case is supported? CC: @nyechiel

@sridhargaddam This is exactly what the setup would look like. I would even be interested in making it working with only one local cluster behind a NAT (home network) and a remote one, publicly accessible, which would stand as a broker.

@sridhargaddam This is exactly what the setup would look like. I would even be interested in making it working with only one local cluster behind a NAT (home network) and a remote one, publicly accessible, which would stand as a broker.

The problem is that both your clusters are behind the same NAT-Router (to the internet) and do not have directly ip-reachability. So, it's like trying to connect two OnPrem clusters where none of them have a public-ip (or externally reachable IP). Unfortunately, we cannot establish IPsec tunnels in such setup.

What is possible with Submariner is: Scenario-1: Say, you have two clusters, one is onPrem and the other one is on Public cloud. In this scenario, the GatewayNode on Public cloud should have a public-ip so that OnPrem cluster GatewayNode can connect to the public-ip. IPsec supports NAT-T, so onPrem cluster need NOT have public-ip and can be behind a NAT Router.

Scenario-2: If you have three clusters, one is onPrem and the other two are on Public cloud. In this scenario, the Gateway nodes on each of the public cloud clusters need a public-ip.

Scenario-3: If you have two clusters, both of them are onPrem and have IP reachability. In this scenario (aka --disable-nat), you don't need any public-ips on your clusters as the GatewayNode on one cluster can directly connect to a GatewayNode on another cluster.

If you make one of your clusters as publicly accessible (+Broker), it would be like Scenario-1 and should work.

@sridhargaddam Thanks a lot for your time. I'll definitely give a try to both the topologies tomorrow. It should be enough for my needs.

Have a great end of the day

submariner-io / submariner

Pods communication not working when running locally #761