search

submariner-io / submariner

Networking component for interconnecting Pods and Services across Kubernetes clusters.
https://submariner.io
Apache License 2.0
2.41k stars 190 forks source link

Submariner `subctl join cluster` not working in aws eks cluster with default vpc cni #2265

Closed pkbhowmick closed 1 year ago

pkbhowmick commented 1 year ago

What happened:

  1. Installed eks cluster using eksctl! 
    apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: submariner-1
region: us-east-1
nodeGroups:
- name: ng-1
instanceType: t3.medium
desiredCapacity: 3
  2. Deploy broker using subctl. 
    ~ $ subctl deploy-broker --globalnet
✓ Setting up broker RBAC 
✓ Deploying the Submariner operator 
✓ Created operator CRDs
✓ Created operator namespace: submariner-operator
✓ Created operator service account and role
✓ Created submariner service account and role
✓ Created lighthouse service account and role
✓ Deployed the operator successfully
✓ Deploying the broker 
✓ Saving broker info to file "broker-info.subm" 
✓ Backed up previous file "broker-info.subm" to "broker-info.subm.2023-01-30T11_49_46+06_00"

    Operator pod:

    ~ $ kubectl get pods -n submariner-operator
NAME                                   READY   STATUS    RESTARTS   AGE
submariner-operator-5997cd5654-2h58x   1/1     Running   0          101s

    There was an red line in operator log:

    2023-01-30T05:49:40.977Z INF ..s/metrics/metrics.go:54 cmd                  Install prometheus-operator in your cluster to create ServiceMonitor objects error="no ServiceMonitor registered with the API"
  3. Try to join into the broker: 
    
~ $ subctl join broker-info.subm --globalnet
✓ broker-info.subm indicates broker is at <aws-server-address>
✓ Discovering network details 
    Network plugin:  generic
    Service CIDRs:   [10.100.0.0/16]
    Cluster CIDRs:   []
? What's the Pod CIDR for your cluster? 192.168.0.0/16
✓ Retrieving the gateway nodes 
✓ Retrieving all worker nodes 
? Which node should be used as the gateway? ip-192-168-16-243.ec2.internal
✓ Labeling node "ip-192-168-16-243.ec2.internal" as a gateway 
✓ Gathering relevant information from Broker 
✓ Retrieving Globalnet information from the Broker 
✓ Validating Globalnet configuration
✓ Assigning Globalnet IPs
✓ Allocated global CIDR 242.0.0.0/16
✓ Updating the Globalnet information on the Broker 
✓ Deploying the Submariner operator 
✓ Created operator namespace: submariner-operator
✓ Created operator service account and role
✓ Created submariner service account and role
✓ Created lighthouse service account and role
✓ Creating SA for cluster 
✓ Connecting to Broker 
✓ Deploying submariner 
✓ Submariner is up and running
But it can't see any gateway, route agent, light house etc pod there. I can see some error logs in the operator pod:

2023-01-30T05:57:14.862Z ERR ..oller/controller.go:326 Reconciler error error="error discovering cluster network: error listing the Daemonsets: daemonsets.apps is forbidden: User \"system:serviceaccount:submariner-operator:submariner-operator\" cannot list resource \"daemonsets\" in API group \"apps\" in the namespace \"kube-system\"" controller=submariner-controller controllerGroup=submariner.io controllerKind=Submariner name=submariner namespace=submariner-operator reconcileID=0f713a5d-5432-4209-8949-4604b6b4e3b2 submariner={"name":"submariner","namespace":"submariner-operator"}

After running `subctl join` command again. It was showing some timeout error:

~ $ subctl join broker-info.subm --globalnet ✓ broker-info.subm indicates broker is at ✓ Discovering network details Network plugin: generic Service CIDRs: [10.100.0.0/16] Cluster CIDRs: [] ? What's the Pod CIDR for your cluster? 192.168.0.0/16 ⢄⡱ Retrieving the gateway nodes There are 1 node(s) labeled as gateways:

subctl version: v0.14.1


**What you expected to happen**:
After join operation other necessary pods should be installed in the server.
**How to reproduce it (as minimally and precisely as possible)**:
Described above
**Anything else we need to know?**:
No
**Environment**:
- Diagnose information (use `subctl diagnose all`):

Cluster "submariner-1.us-east-1.eksctl.io" ✓ Checking Submariner support for the Kubernetes version ✓ Kubernetes version "v1.22.17-eks-e0e89e9" is supported

✗ Checking Submariner support for the CNI network plugin ✗ The detected CNI plugin ("") is not supported by Submariner. Supported plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico kindnet] ✗ Checking gateway connections ✗ No gateways were detected ✓ Globalnet deployment detected - checking if globalnet CIDRs overlap ✓ Clusters do not have overlapping globalnet CIDRs ✗ Checking Submariner pods ✗ Error obtaining Daemonset "submariner-gateway": daemonsets.apps "submariner-gateway" not found ✗ Error obtaining Daemonset "submariner-routeagent": daemonsets.apps "submariner-routeagent" not found ✗ Error obtaining Deployment "submariner-lighthouse-agent": deployments.apps "submariner-lighthouse-agent" not found ✗ Error obtaining Deployment "submariner-lighthouse-coredns": deployments.apps "submariner-lighthouse-coredns" not found ✗ Error obtaining Daemonset "submariner-globalnet": daemonsets.apps "submariner-globalnet" not found ✗ Error obtaining Daemonset "submariner-metrics-proxy": daemonsets.apps "submariner-metrics-proxy" not found ✓ Checking Submariner support for the kube-proxy mode ✓ The kube-proxy mode is supported ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed ✗ Unable to obtain the local endpoint: endpoints.submariner.io "local Endpoint" not found ✗ Checking Globalnet configuration ✗ Found 0 ClusterGlobalEgressIP resources but only the default instance ("cluster-egress.submariner.io") is supported ✗ Couldn't find the default ClusterGlobalEgressIP resource("cluster-egress.submariner.io")

⚠ Service discovery is not installed

- Gather information (use `subctl gather`):

~ $ subctl gather Cluster "submariner-1.us-east-1.eksctl.io" Gathering information from cluster "submariner-1.us-east-1.eksctl.io" ✓ Gathering connectivity logs ✓ Found 0 pods matching label selector "app=submariner-gateway" ✓ Found 0 pods matching label selector "app=submariner-routeagent" ✓ Found 0 pods matching label selector "app=submariner-globalnet" ✓ Found 0 pods matching label selector "app=submariner-networkplugin-syncer" ✓ Found 0 pods matching label selector "app=submariner-addon" ✓ Gathering connectivity resources ✓ Gathering CNI data from 0 pods matching label selector "app=submariner-routeagent" ✓ Gathering CNI data from 0 pods matching label selector "app=submariner-gateway" ✓ Gathering globalnet data from 0 pods matching label selector "app=submariner-globalnet" ✓ Gathering cable driver data from 0 pods matching label selector "app=submariner-gateway" ✓ Found 0 endpoints in namespace "submariner-operator" ✓ Found 0 clusters in namespace "submariner-operator" ✓ Found 0 gateways in namespace "submariner-operator" ✓ Found 0 clusterglobalegressips in namespace "" ✓ Found 0 globalegressips in namespace "" ✓ Found 0 globalingressips in namespace "" ⚠ Gathering service-discovery logs ⚠ The Submariner service discovery components are not installed ⚠ Gathering service-discovery resources ⚠ The Submariner service discovery components are not installed ✓ Gathering broker logs ✓ Gathering broker resources ✓ Found 0 endpoints in namespace "submariner-k8s-broker" ✓ Found 0 clusters in namespace "submariner-k8s-broker" ✓ Found 0 endpointslices by label selector "endpointslice.kubernetes.io/managed-by=lighthouse-agent.submariner.io" in namespace "submariner-k8s-broker" ✓ Found 0 serviceimports in namespace "submariner-k8s-broker" ✓ Gathering operator logs ✓ Found 1 pods matching label selector "name=submariner-operator" ✓ Gathering operator resources ✓ Found 1 submariners in namespace "submariner-operator" ✓ Found 0 servicediscoveries in namespace "submariner-operator" ✓ Found 1 deployments by field selector "metadata.name=submariner-operator" in namespace "submariner-operator" ✓ Found 0 daemonsets by label selector "app=submariner-gateway" in namespace "submariner-operator" ✓ Found 0 daemonsets by label selector "app=submariner-routeagent" in namespace "submariner-operator" ✓ Found 0 daemonsets by label selector "app=submariner-globalnet" in namespace "submariner-operator" ✓ Found 0 deployments by label selector "app=submariner-networkplugin-syncer" in namespace "submariner-operator" ✓ Found 0 deployments by label selector "app=submariner-lighthouse-agent" in namespace "submariner-operator" ✓ Found 0 deployments by label selector "app=submariner-lighthouse-coredns" in namespace "submariner-operator" Files are stored under directory "submariner-20230130060715/submariner-1.us-east-1.eksctl.io"


- Cloud provider or hardware configuration: Kuberenetes version 1.22
- Install tools: subctl version: v0.14.1
- Others:
tpantelis commented 1 year ago

This looks like the same issue as https://github.com/submariner-io/submariner-operator/issues/2450 which will be in 0.14.2.

pkbhowmick commented 1 year ago

This looks like the same issue as submariner-io/submariner-operator#2450 which will be in 0.14.2.

Is it fixed? Will be available in next release?

skitt commented 1 year ago

This looks like the same issue as submariner-io/submariner-operator#2450 which will be in 0.14.2.

Is it fixed? Will be available in next release?

As @tpantelis said, it will be available in 0.14.2. It was fixed by https://github.com/submariner-io/submariner-operator/pull/2451 which will be included in 0.15.0 and 0.14.2.

13567436138 commented 1 year ago

when will 0.15.0 be released,I face the same problem