Submariner compatibilty

[LavredisG]

Is kindnet compatible with Submariner or I need another CNI? I joined a cluster even using my own clustercidr and servicecidr as flags on the subctl join command, but I get no gateways/endpoints or CIDRs:

[stmcginnis]

I don't know much about submariner, but I would assume it is not compatible with kind and the kindnet that is created.

In normal operation, you need to set up ingress or use cluster-provider-kind to enable access to the cluster. The typical use case for a kind cluster is for local development and testing, but cross-cluster networking.

cloud-provider-kind gets you part of the way there, but that only exposes Services via LoadBalancer. Anything that expects to have access to both the pod and service network will not be able to reach the expected network.

[aojea]

I know some time ago submariner used kind in their CI system, @skitt as maintainer of the project may know better

[skitt]

Submariner still uses kind in its CI, and supports it for testing and development; see https://submariner.io/getting-started/quickstart/kind/ for details. I can’t remember off-hand how the kind setup differs from the defaults. If you follow the quickstart guide you’ll end up with something like this:

Cluster "cluster1"
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS     GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS   
submariner-k8s-broker   submariner-broker   connectivity   no          242.0.0.0/8      65536                                      

 ✓ Showing Connections
GATEWAY           CLUSTER    REMOTE IP    NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.     
cluster2-worker   cluster2   172.18.0.6   no    libreswan      100.67.0.0/16, 10.131.0.0/16   connected   122.425µs    

 ✓ Showing Endpoints
CLUSTER    ENDPOINT IP   PUBLIC IP      CABLE DRIVER   TYPE     
cluster1   172.18.0.7    82.65.25.201   libreswan      local    
cluster2   172.18.0.6    82.65.25.201   libreswan      remote   

 ✓ Showing Gateways
NODE              HA STATUS   SUMMARY                               
cluster1-worker   active      All connections (1) are established   

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  kindnet
        Service CIDRs:   [100.66.0.0/16]
        Cluster CIDRs:   [10.130.0.0/16]

 ✓ Showing versions 
COMPONENT                  REPOSITORY           CONFIGURED   RUNNING              ARCH    
submariner-gateway         quay.io/submariner   devel        devel-44272f008bee   amd64   
submariner-routeagent      quay.io/submariner   devel        devel-44272f008bee   amd64   
submariner-metrics-proxy   quay.io/submariner   devel        devel-615bd933654c   amd64   
submariner-operator        quay.io/submariner   devel        devel-bf47293142c7   amd64   

[LavredisG]

Thank you for your answers. I actually had followed the quickstart guide before creating the issue, with the difference that I didn't use the submariner scripts to create the clusters. I will search the source code a bit to see if I can find anything besides the default cni being installed.

[skitt]

Right, it would be nice if kind create cluster --name=cluster1 && kind create cluster --name=cluster2 && subctl deploy-broker --context=kind-cluster1 && subctl join --context=kind-cluster1 broker-info.subm && subctl join --context=kind-cluster2 broker-info.subm just worked :wink:.

[skitt]

The main thing the setup code does is update the kubeconfig generated by kind to use an externally-accessible IP address (instead of 127.0.0.1) — without that, the Submariner components won’t know how to connect to the broker.

The setup also ensures that the kind clusters use non-overlapping subnets, but it should be possible to use conflicting subnets with deploy-broker --globalnet.

[BenTheElder]

to use an externally-accessible IP address (instead of 127.0.0.1) — without that, the Submariner components won’t know how to connect to the broker.

This is not something we'd likely make the defaults, changing kind to be externally accessible by default would have huge implications and is something we actively discourage in our docs: https://kind.sigs.k8s.io/docs/user/configuration/#api-server

However, you can do this by just configuring kind, even if discouraged.

[skitt]

This is not something we'd likely make the defaults, changing kind to be externally accessible by default would have huge implications and is something we actively discourage in our docs: https://kind.sigs.k8s.io/docs/user/configuration/#api-server

Oh yes indeed, I didn’t mean to imply kind should be changed to accommodate Submariner, far from it. In fact as far as I’m concerned this issue should be closed, and perhaps a corresponding documentation issue opened in the Submariner project itself.