search

submariner-io / submariner

Networking component for interconnecting Pods and Services across Kubernetes clusters.
https://submariner.io
Apache License 2.0
2.36k stars 184 forks source link

Submariner 0.12.0 not able to establish connection between sites. Error: The detected CNI network plugin ("") is not supported by Submariner. #3042

Open BhavaniYalamanchili opened 3 weeks ago

BhavaniYalamanchili commented 3 weeks ago

ISSUE:

The Submariner is not able to establish the connection between the sites. The error we are seeing in the diagnose all command is Error Message: The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico]

SETUP:

Site 1: OCP 4.10 Site 2: OCP 4.10 Submariner version: v0.12.0

We are using OVNKubernetes From the output of oc get network/cluster -o yaml We can see network type as this networkType: OVNKubernetes

Outputs of show all and diagnose all commands

Show all for Site 1

Cluster "site1"
 ✓ Detecting broker(s)
NAMESPACE                NAME                     COMPONENTS
submariner-k8s-broker    submariner-broker        service-discovery, connectivity

 ✓ Showing Connections
GATEWAY                          CLUSTER  REMOTE IP       NAT  CABLE DRIVER  SUBNETS                       STATUS  RTT avg.
control-1-ru2.ocp-psb-01.gbbper  site2    192.168.126.30  no   libreswan     172.31.0.0/16, 10.132.0.0/14  error   0s

 ✓ Showing Endpoints
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE
site1                         192.168.54.30   192.168.54.30   libreswan           local
site2                         192.168.126.30  192.168.126.30  libreswan           remote
site1                         192.168.54.31   192.168.54.31   libreswan           local
site1                         192.168.54.32   192.168.54.32   libreswan           local

 ✓ Showing Gateways
NODE                            HA STATUS       SUMMARY
control-1-ru2.ocp-psa-01.gbbper active          0 connections out of 1 are established
control-1-ru3.ocp-psa-01.gbbper passive         There are no connections
control-1-ru4.ocp-psa-01.gbbper passive         There are no connections

    Discovered network details via Submariner:
 ✓ Showing Network details
        Network plugin:
        Service CIDRs:   []
        Cluster CIDRs:   []

 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0

Show all for Site 2

Cluster "local-config"
 ✓ Detecting broker(s)

 ✓ Showing Connections
GATEWAY                          CLUSTER  REMOTE IP      NAT  CABLE DRIVER  SUBNETS                       STATUS     RTT avg.
control-1-ru2.ocp-psa-01.gbbper  site1    192.168.54.30  no   libreswan     172.30.0.0/16, 10.128.0.0/14  connected  988.791µs

 ✓ Showing Endpoints
CLUSTER ID                    ENDPOINT IP     PUBLIC IP       CABLE DRIVER        TYPE
site2                         192.168.126.30  192.168.126.30  libreswan           local
site1                         192.168.54.30   192.168.54.30   libreswan           remote
site2                         192.168.126.31  192.168.126.31  libreswan           local
site2                         192.168.126.32  192.168.126.32  libreswan           local

 ✓ Showing Gateways
NODE                            HA STATUS       SUMMARY
control-1-ru2.ocp-psb-01.gbbper active          All connections (1) are established
control-1-ru3.ocp-psb-01.gbbper passive         There are no connections
control-1-ru4.ocp-psb-01.gbbper passive         There are no connections

    Discovered network details via Submariner:
 ✓ Showing Network details
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
service-discovery               quay.io/submariner                                    0.12.0
COMPONENT                       REPOSITORY                                            VERSION
submariner                      quay.io/submariner                                    0.12.0
submariner-operator             quay.io/submariner                                    0.12.0
service-discovery               quay.io/submariner                                    0.12.0

Diagnose all for Site 1

sh-4.4$ /root/.local/bin/subctl diagnose all --kubeconfig site-1-kubeconfig
Cluster "site1"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✗ Checking Submariner support for the CNI network plugin
 ✗ The detected CNI network plugin ("") is not supported by Submariner. Supported network plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico]

 ✗ Checking gateway connections
 ✗ Connection to cluster "site2" is not established

 ✗ Checking Submariner pods
 ✗ Error obtaining Daemonset "submariner-routeagent": daemonsets.apps "submariner-routeagent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-agent": deployments.apps "submariner-lighthouse-agent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-coredns": deployments.apps "submariner-lighthouse-coredns" not found

 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs

 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported

 ✓ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ The firewall configuration allows metrics to be retrieved from Gateway nodes

 ✗ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✗ The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 172.31.0.0. Please check that your firewall configuration allows UDP/4800 traffic.

 ✓ Globalnet is not installed - skipping

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

Diagnose all for Site 2

sh-4.4$ /root/.local/bin/subctl diagnose all --kubeconfig site-2-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported

 ✓ Checking gateway connections
 ✓ All connections are established

 ✓ Checking Submariner pods
 ✓ All Submariner pods are up and running

 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs

 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported

 ✓ Checking the firewall configuration to determine if the metrics port (8080) is allowed
 ✓ The firewall configuration allows metrics to be retrieved from Gateway nodes

 ✓ Checking the firewall configuration to determine if VXLAN traffic is allowed
 ✓ This check is not necessary for the OVNKubernetes CNI plugin
 ✓ The firewall configuration allows VXLAN traffic

 ✓ Globalnet is not installed - skipping

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.
yboaron commented 2 weeks ago

A. You can read how Submariner detects if CNI is OVN-K8S here (in 0.12 release)

B. I can see that OVN-K8S was detected successfully for site2 , both site1 and site2 are running OCP 4.10, is there any difference in OVN-K8S configuration between these clusters ?

dfarrell07 commented 2 weeks ago

FYI, SubM 0.12 is a very old version that's long since not supposed to be supported. If you get a chance to update, that would likely be a very good idea.

BhavaniYalamanchili commented 2 weeks ago

@yboaron Based on the function code, I have listed out some oc commands to verify the pods and services the function is fetching and some output related to submariner CR Here are the outputs:

  1. oc get pods -A -1 name=ovnkube-db 
    No resources found

    On both sites, it is the same output

  2. oc get pods -n openshift-ovn-kubernetes Site 1 
    NAME                   READY   STATUS    RESTARTS   AGE
ovnkube-master-42r22   6/6     Running   34         325d
ovnkube-master-fbn4v   6/6     Running   6          37d
ovnkube-master-kbs2d   6/6     Running   6          37d
ovnkube-node-7nxcq     5/5     Running   30         325d
ovnkube-node-bcshb     5/5     Running   267        325d
ovnkube-node-br4cf     5/5     Running   25         325d
ovnkube-node-bsb2b     5/5     Running   41         325d
ovnkube-node-dpt65     5/5     Running   30         324d
ovnkube-node-fb7rw     5/5     Running   57         325d
ovnkube-node-g6jsc     5/5     Running   25         325d
ovnkube-node-jkdt8     5/5     Running   30         325d
ovnkube-node-qllh7     5/5     Running   25         325d
ovnkube-node-rb2bt     5/5     Running   25         324d
ovnkube-node-tc4qq     5/5     Running   5          37d
ovnkube-node-w7rzd     5/5     Running   25         325d

    Site 2

    NAME                   READY   STATUS    RESTARTS       AGE
ovnkube-master-ccnvn   6/6     Running   4 (4d3h ago)   4d3h
ovnkube-master-sfjvh   6/6     Running   4 (4d3h ago)   4d3h
ovnkube-master-xqbxp   6/6     Running   0              4d3h
ovnkube-node-48rwm     5/5     Running   0              4d3h
ovnkube-node-852x6     5/5     Running   0              4d3h
ovnkube-node-88kqh     5/5     Running   0              4d3h
ovnkube-node-bn5s6     5/5     Running   0              4d3h
ovnkube-node-frd5g     5/5     Running   0              4d3h
ovnkube-node-g6fc4     5/5     Running   0              4d3h
ovnkube-node-hx8hb     5/5     Running   0              4d3h
ovnkube-node-kp6l8     5/5     Running   0              4d3h
ovnkube-node-ld7xj     5/5     Running   0              4d3h
ovnkube-node-n89hp     5/5     Running   0              4d3h
ovnkube-node-pbmpk     5/5     Running   0              4d3h
ovnkube-node-tv8sc     5/5     Running   0              4d3h
  3. oc get services -n openshift-ovn-kubernetes | grep ovnkube-db Site 1 
    ovnkube-db              ClusterIP   None         <none>        9641/TCP,9642/TCP   325d

    Site 2

    ovnkube-db              ClusterIP   None         <none>        9641/TCP,9642/TCP   299d
  4. oc get submariner submariner -n submariner-operator -o yaml Site 1 - CR in Site 1 is showing till the Spec only, I am wondering why the status is not shown. 
    # oc get submariner submariner -n submariner-operator -o yaml
apiVersion: submariner.io/v1alpha1
kind: Submariner
metadata:
creationTimestamp: "2024-06-06T13:22:34Z"
finalizers:
- controllers.submariner.io/cleanup
generation: 1
managedFields:
- apiVersion: submariner.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
  f:spec:
    .: {}
    f:broker: {}
    f:brokerK8sApiServer: {}
    f:brokerK8sApiServerToken: {}
    f:brokerK8sCA: {}
    f:brokerK8sRemoteNamespace: {}
    f:brokerK8sSecret: {}
    f:ceIPSecDebug: {}
    f:ceIPSecIKEPort: {}
    f:ceIPSecNATTPort: {}
    f:ceIPSecPSK: {}
    f:ceIPSecPSKSecret: {}
    f:clusterCIDR: {}
    f:clusterID: {}
    f:connectionHealthCheck:
      .: {}
      f:enabled: {}
      f:intervalSeconds: {}
      f:maxPacketLossCount: {}
    f:debug: {}
    f:namespace: {}
    f:natEnabled: {}
    f:repository: {}
    f:serviceCIDR: {}
    f:serviceDiscoveryEnabled: {}
    f:version: {}
manager: subctl
operation: Update
time: "2024-06-06T13:22:34Z"
- apiVersion: submariner.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
  f:metadata:
    f:finalizers:
      .: {}
      v:"controllers.submariner.io/cleanup": {}
manager: submariner-operator
operation: Update
time: "2024-06-06T13:25:22Z"
name: submariner
namespace: submariner-operator
resourceVersion: "2020929472"
uid: e6f8ad61-63bc-4d8a-8e56-2d7ad292a2d1
spec:
broker: k8s
brokerK8sApiServer: api.ocp-psa-01.gbbper.priv:6443
brokerK8sApiServerToken: eyJhbGciOiJSUzI1NiIsImtpZCI6IngwR2pZY2NWTXJtRlI2T3pxVjFzSHU5OU5SNm9fUkt3V1BMTFJxcnV3LVkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2x1c3Rlci1zaXRlMS10b2tlbi1ma2NqciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbHVzdGVyLXNpdGUxIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZTYxOTM4ZTItNDgxMC00MGZkLTlmMTctZGQ2NDg3NTM5NThkIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpjbHVzdGVyLXNpdGUxIn0.Z_KAnNtLYHlPRjm7x5ZSwWLAoMyd3Pq_Z9z4LD3iwxrmgfExt5iltotssdeQXDkYvOHYWL7f5XTT7FcvBhKt57aQ9s5tDvVz1LIaehwnMnHwwdYadhSDBgfvrEbAmCByxx0CXOmejuBS5sov5oIhJSeSZzT1wJ6xVsEg-pyQbewb1All0oQIaEmtQtzpWg4OeL2WAxf2u-SG2Er8RbWj_EZ5fwgAHwU2RabvN59-JhA1jE9ZoK_4fkUNlmajDYHr8l_LwtDd4ScmY7hTBARBH6uTkTx3H787qsp7zHgSDxhofT1M51ABdhea3WaWGjqefb-nWOOenSl9KyAVyqF-b_dOD_6jP3pktDE2BPD0Aj2ehOd6FoGorL7ZyYbo1-oHJusC1pE1B46K8Ij7eI_6iSkds5etONhwVonjlPt_L8vdIqn9rBaVJhHbEwz1sHrHvkKd9G0Ka41SLEjZtI7TUkTWTGUF41ViAF2D4OGIYyuiJO9YIUQu-a1tTt8bLpu-_67DHYHA2jZh510aRohUgKthUxKC7E4atVlSwYZk5Lm-5r4kscRb6r2SqIxy2EOif1dtcnpiECYgnvBFtGa1MRbna_f2VRh6KXlOWrtzAZmFIe2pE6BftIidknync1ZAXfJz-zzNsQ_YFVBzsGqiQnfUn_Opr9VWI7TSSgIgYgo
brokerK8sCA: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0lJZkRtandjUGF1K1F3RFFZSktvWklodmNOQVFFTEJRQXdOekVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TRXdId1lEVlFRREV4aHJkV0psTFdGd2FYTmxjblpsY2kxc1lpMXphV2R1WlhJdwpIaGNOTWpNd056SXdNVFV3TVRFMFdoY05Nek13TnpFM01UVXdNVEUwV2pBM01SSXdFQVlEVlFRTEV3bHZjR1Z1CmMyaHBablF4SVRBZkJnTlZCQU1UR0d0MVltVXRZWEJwYzJWeWRtVnlMV3hpTFhOcFoyNWxjakNDQVNJd0RRWUoKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGNiT0dQbnRaTE1sR2xPQ1FHODFnWCt6dG1FMUFZWgp2RUkybFM4UCtCOGVkSDJCdk9GYkc2dWc1WUhSN1h4RkdlNVllSjZ4Tkw5M01OS05uK1BLcGhrNm5kdlJDTjZtCkF0RFI3WnBBczFYVGZEOHNHOHg3TWFYZTF5RU9SdmtZQXBSLzVOSFFhSnZzUWxqaXY3blQvUWNLTFRKbXpBUUsKS0ZwQ1pLd3NVc1R0NEtsYXVPLzlDWnZpMVZlcTV0NlJhTE1xdjZyREREUC8zcHo4U0NzTlYvR3hIeGhXVWl4dQpVK2RxbmF2RmxGa0xOdys0Qnh1SnFpcUJHNGNwV2p5cjFEMmJBcThvVnoxN1JaTmw2M09VdUxGa3VYdWlsUVBmCmkzSXN1WDQ2WTE0T01icHRzTTltNzZmdElSUWJNU0paZWxqdmlqVG9VKzZ2bFRNak0vUDdkQ0VDQXdFQUFhTkMKTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkEwRwpyM2FvSk1UdVpmR3JjZlQrYm94NFZpQkdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5dkRWRlpFOXhpUzRxCndaQS9DaFRjOG9IMDlCZmlBNTlpNjd0cldLMzZVVGo0SFZiM2ZMOEFXeXBnM2tZSGh2azZ2cUJnRHVmVFpOUjcKU2Y1OS9SY0w1VVRSdTA4bzlETHNvSTlFWUxkMjlRcTdjdHBsUmw1RmZ3UjY5ZjlOL2kwOFlRZDVKLzBLNHA1OAorR1M0WWhTR1d6c2oxUnZ2MUJBTkY2a056MFhGOG1lSVJVWkcyZFBBaFoyVWRxMnBPQ3d2MmFqZHZqWk14ZFBFCkdPMStSbDdBa3hNTEhKbXZDOGJVbEM2U3hsYm1BV2FiaTY5M3VkWjZUQkJwTjRwUkFNZnQvOHFzM3ZsRTlpMzcKajFTWitMTVNQR2xINWlwMUMzbjlDNlUxRlpBQU9vN09uYXJaSk9yZ09ua2h4MVBwM09wTDlYY1I3Q0Y2eE5sMQowbWtYcnY5aQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEUURDQ0FpaWdBd0lCQWdJSUVsSU52VkpFRVBVd0RRWUpLb1pJaHZjTkFRRUxCUUF3UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNQjRYRFRJek1EY3lNREUxTURFeE5Gb1hEVE16TURjeE56RTFNREV4TkZvd1BqRVNNQkFHQTFVRQpDeE1KYjNCbGJuTm9hV1owTVNnd0pnWURWUVFERXg5cmRXSmxMV0Z3YVhObGNuWmxjaTFzYjJOaGJHaHZjM1F0CmMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2Qy82VVNrZEhkT3MKb2NuMFBLdm52RUtEcTZ3bzRkNXJMVGUybG5jam5UVWlRQUFtNkd5YkJRVC9BNTZyb3kxSUpPNGtpWXlscFFBRQpQaEk0dkIzMGdxNjZLS0czdlpkc3dpemt6OTFEVE9wV05ZY3Bmbzh4R3ZzZFZSM1RudVNnREdCVHV3NGNCYzV3Ci9LRXlnS3BtZk1WenJiS0cxbnlNY1daUHhBTkM4V3kyRm1NTSt1eVFFaVRCak9mVVBvWTZkZVVqUXpNZkp5dGsKR1NjMWordEUwejFsWkRZMjJQMW5yTDh3TkR3Y0R3T0gydGcvcHhVV2VkeEQ0SGFLMGwzRjQvOVlxYW56T21oRQo5Q2dRaEtHTGdERzUxUWE0WU1qWUl2bTJqUExvNUpOaU05SjlkZzhKSEdxNm5SM2hkZmI4TkhjY1B4bWRtRzhjClEwWVlYUWMwcXdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVVNBZzgvY29iR1lUaitDb0xtMGdaaW9TUnljd3dEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUtDYkVIN2lKQVN1ajRVcHo0ajd4RVBzY05RS0tOS3RzQ29TbHpuWTBuRTVYUjhwV3VVdmxvTS9xdVVQClUrL3RSWldTaXV3SFdEeUJHY2tubUJhOS92aE1HVzNNcjlCOWxlL0srVHpSQlg0aGF6YWRDbXU2b0p6K1lxWTcKRkQ4elVqYkxZN3BBQ1BFdkpJY3FRWTJTRDdlODd5N0Jnam5UM3lUMzdyd01pcXV6bHI1QmNjLzg3WjRCanNnTAo4MjAyUXVFYml6Q3ZpTnBqYUNyamw4YWtKTG1SRnZZVERiaTk0bzh1YklyVHZ6a1NpbmNhWnQ5eHlibWhySEt4CmhPWjd3Tnh4VzNQRUU5RVZRN1c3RVhLYUhadk1sVWZNUXVrSS84Y05uWDltdm9wSUY5MEVrb3hkd2JQZ00yZ3gKVXN5TllpRENkWENVcGFJWUJFNTUrbWQyb3pJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEVERDQ0FqU2dBd0lCQWdJSVc4K0ZZTXhsdklJd0RRWUpLb1pJaHZjTkFRRUxCUUF3UkRFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyYVdObExXNWwKZEhkdmNtc3RjMmxuYm1WeU1CNFhEVEl6TURjeU1ERTFNREV4TkZvWERUTXpNRGN4TnpFMU1ERXhORm93UkRFUwpNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyCmFXTmxMVzVsZEhkdmNtc3RjMmxuYm1WeU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQXhLTWdMcmtyc2dSaE4zZEY1TFlBZForeFdmZXNJR0laa0FVZ25tV1Y3azdVbjJCUVkvbUNuekVPTHhibgpDNm56MWFKT3R1SDNoZWg4bE5MVGIxc1JQZkFpaVBIdVBOaDRrZXMxcDdiajVHcXd2MmRNc0tESUw2aHAxbE4xClJEWFNiZUw1ajd6UjFXbUJrL01taUQ2L2wwaWpIZGRjMXpRNmlFQ2p6MXFsbHl6UDM1ZW1VS1dwcG5iQld0S2IKYjJXNGlEdVhBQ0JvbWgzVTBkalZWYitYR3JiQkhaQ21iaWMrRWcvYi8ya3J2WFpZTG93N0QwTjVaRkRkZG84TgpRcFNabTVaOUZYNHE0LzJqMjVZUjAzSGNFd1RwNVI3UXFsWDYxWUZuT2dJWUNQNzB2d0N5aXpVclE4S3dnNGZ6ClZDaGl5Q1pCSXlHYzUrR3hzcUViZXVmL2VRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WUQKVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVGamJyS3Y0NTFiWmgvaDM1SU9hREtuUnBpd013RFFZSgpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKK2kzVURUN3Y1cVFpS1gzSW96Qk5MVzBiN3Fncy9tUjJueGFnOWo1RFFuCkZUelRGYjl6dDVUNG5VQ0hwdVRtdjZudWFvUWhrL1BUV2RDV21uSXp5WmVjMERxcCtWWXFCQnFyZGFKUU00b0oKVFdxQTU1WFZGU0s2aVpWS09wNDNETHg2aVhMN1NEWXNiQ000d1I2a3dSbEtib3Z3NjN4SE42QWtWcCtQRzk5dQo3RkR4REU5RG00SW1zd3hhcGFEMnFZTlhkdjRpS1R5cmtHWUg2elM1VmQwSUxtSkFaQjZxMk5sUnpEb2tSY3NTCkovRUJwU3pjVVFuUlRjbkkxM21VMTFqelZRVFRMYktsVXV2d01LU3ZPa0c5bXVsU1p3SUc0RnR5SVd5d1dGbjEKQ093ZUNlaTh5K0FpaE5BYkdET3U1Q0h6WGl3dDVxYko3T3hCeGY2RUh6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRGx6Q0NBbitnQXdJQkFnSUlNRmpCOEN4eWczWXdEUVlKS29aSWh2Y05BUUVMQlFBd1dURlhNRlVHQTFVRQpBd3hPYjNCbGJuTm9hV1owTFd0MVltVXRZWEJwYzJWeWRtVnlMVzl3WlhKaGRHOXlYMnh2WTJGc2FHOXpkQzF5ClpXTnZkbVZ5ZVMxelpYSjJhVzVuTFhOcFoyNWxja0F4TmpnNU9EWTNPRFV6TUI0WERUSXpNRGN5TURFMU5EUXgKTWxvWERUTXpNRGN4TnpFMU5EUXhNMW93V1RGWE1GVUdBMVVFQXd4T2IzQmxibk5vYVdaMExXdDFZbVV0WVhCcApjMlZ5ZG1WeUxXOXdaWEpoZEc5eVgyeHZZMkZzYUc5emRDMXlaV052ZG1WeWVTMXpaWEoyYVc1bkxYTnBaMjVsCmNrQXhOamc1T0RZM09EVXpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTNCbTUKckxjdS9kcExrczZNcS90TGxvbjJTMFM2aXNzcEZSb1FRZjNwZXg2WDNHN0NLOWpQYldsQ0dWeElpKzdqTW9XYQpHYTYwc3hmeWZ6L0NDMlZpWnJBNUMrUU1GeTJoc29kalFxeEVDNWNVVERmaHRiVzVqYWltY25xUUgrQmRXeHhECmN5c0NWQVNaWFZ4WmRvaVhqTXAwb3A2QWkydzBZekE3UzJDSWFKaTBsTjNBamNFalFRWnNPTEJiMGNrQjFrWHkKNXJMWGI3cTR0WkxQVmtzVm94dncrOUFDNzdxdkNreGwzYWd6TEowZGRjYnFXUjEvS29OdHlRMklpb0pWVmtregoyUHB2ekJ1S3hYUWR0d2s5dWwzWWhwYWIvMEhxM2VjYlpyUGZmY2VVa3RZUjc2cW9RUzhBL0tQNFI1SG5RN3VZClJGUDdQbE9hS3BqdUNqVTVuUUlEQVFBQm8yTXdZVEFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC8KQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0h3WURWUjBqQkJndwpGb0FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLS2o4NW9MClU5a1JLdDlWRVdSNVp4UmdIYzc5TDkrckhYTldxVGpRV2xUcEhrd1B5RE5wLy83OHpUdmNLbkREaEZ1dG0yTE8KMHZCbnlBNlZ3QmdWQS9hR1BnUENGUkxRZk9ndEtHNVFhTW8zUC9ZR3RyN0NHb1NpYUFPblRTZjRybjFlVy9oUAovOGlOZEFzZzVJYU5IMFkrcGM1dEVkUDhUOUNpQmUwUDViNGk3ZFNidXVBVFJOK2U0b2ZvRjV5dEd3RHJ5RjZqCm9MSi9ZUHZFSEtWanhxUWpCYkltdzVVMHJMSWpSZEQ3dlMybmNEWm9YY3Q1djBzeUxuN3JabWd0aWJJRkpFYTAKVng5c2RETkZGSWxWcHBYVXU4RnJ2RHY4TUdsNk9CQUVSZXBCQjhndVpsS0R5TlRNaU1DM3hRTjB0N2wxaXNGagprSWN2ZFFuL1dnV3I2cmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVtVENDQTRHZ0F3SUJBZ0lDQzI0d0RRWUpLb1pJaHZjTkFRRUxCUUF3Z2JjeEN6QUpCZ05WQkFZVEFrbFUKTVE4d0RRWURWUVFJRXdaTmIyUmxibUV4RnpBVkJnTlZCQWNURGtObGJuUnlieUJ6WlhKMmFYcHBNUlF3RWdZRApWUVFLRXd0SGNuVndjRzhnUWxCRlVqRVdNQlFHQTFVRUN4TU5RbEJGVWlCVFpYSjJhV05sY3pFVk1CTUdBMVVFCkRCTU1ZMkVnWW5CbGNpQXlNRE00TVRrd053WURWUVFERXpCSGNuVndjRzhnUWxCRlVpQXRJRk5sY25acGVtbHYKSUdScElFTmxjblJwWm1sallYcHBiMjVsSUVsdWRHVnlibUV3SGhjTk1qTXdNakkyTWpNd01EQXdXaGNOTWpVdwpNakkzTWpJMU9UVTVXakNCcFRFTE1Ba0dBMVVFQmhNQ1NWUXhDekFKQmdOVkJBZ01BazFQTVE4d0RRWURWUVFICkRBWk5iMlJsYm1FeEdUQVhCZ05WQkFvTUVFSlFSVklnUW1GdVkyRWdVeTV3TGtFeE5UQXpCZ05WQkFzTUxGVm0KWmk0Z1UybHpkR1Z0YVNCRWFYQmhjblJwYldWdWRHRnNhU0JsSUVOdmJXMTFibWxqWVhScGIyNXpNU1l3SkFZRApWUVFEREIwcUxtRndjSE11YjJOd0xYQnpZUzB3TVM1blltSndaWEl1Y0hKcGRqQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLdm1MakhlaUx3L2p0d3hUUFcvUTJzSXdjb3RWSDVEcWxGOEtvMjEKUGExWml6T0dFTXk5R0hhNUEvYXZ6clFXdW10RGNVTExqemdRaDB6N29hWDNxSWJMbklRMHllbDU5Y1ZGZ1ZWTwpVSFpYN29kaUlPMzdkNVhuZ014TnBlcG1jWFFsOEEzV3hkVkRxRnlOT295OGRVT1dCWk1ONGhmLzN3ZUVhWVczCis5S0FlY2JLbS8wUEcrdkFJNms1bm1sUmtYK2w4T0tIVit6US9FMkd3bjByT1JHTml2YjFOV0lYYzZpVENqRjQKL1hmb1c2aE9sTzJWNWpnM0cvSlJ6VUg5UFUwRjJML3dielNmUU15MHF4SzZwOUZGNHdPL3ZqNlNuMGN6NlFzdwpYeWI1Rm01NXpIYitONE04cVFsRU9pc01iYmZhME9va3NCaGVCU05reC9sRWg3VUNBd0VBQWFPQnZqQ0J1ekEvCkJnbGdoa2dCaHZoQ0FRMEVNaFl3UjJWdVpYSmhkR1ZrSUdKNUlIUm9aU0JUWldOMWNtbDBlU0JUWlhKMlpYSWcKWm05eUlIb3ZUMU1nS0ZKQlEwWXBNQ2dHQTFVZEVRUWhNQitDSFNvdVlYQndjeTV2WTNBdGNITmhMVEF4TG1kaQpZbkJsY2k1d2NtbDJNQTRHQTFVZER3RUIvd1FFQXdJRXNEQWRCZ05WSFE0RUZnUVVZVGh5ajVSbVhud1NmdEhKCkE3Y3VJQnEwT0Zvd0h3WURWUjBqQkJnd0ZvQVVxV29Oc1FyVXhpLzI5TERDN1JPNkpPTFpnUEF3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFBRmlJUHZGdHJYaFM0UExpNS94bmZkV1k1UDNnZERTZlJMMDRoWjROd2JZSTVXZwpZRklMVnlwTnVLNVNSTk1hSWw3WkFaUytnUkJoQVRCVStIdjNLN083VHlLaC9UQ2tnOVpHZzI2SVhxc0t5UHZ0CmFPSUx5SDFTZkRiYXhPN3RPclhrTTA3eGd6S0hocUhWWlQ5YXpPUnVrVlJaa01nd0gySGpmdWNTV0ZrTis2eDIKdm1HempBQWlKQ1JGY0JwNnRLYTY0b3RGdFRmY2FNYVZVRlllZkRqZ285b2tnWTRqZ2UwWXlXQkErc1hOUWsydgpVWHhrMm1IYitjSlAxS1I5ayswOEt1YXFKYkJCS0s0RC9tWHkwcCtYODdDYWpZdFRxQ0lHSXk4T3JROTFIYUxiClhYQzlSMWRYV2krMkdrTmI0SkVhKyt0aFVCaVNsQldycHpycURPUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
brokerK8sRemoteNamespace: submariner-k8s-broker
brokerK8sSecret: broker-secret-66fps
ceIPSecDebug: false
ceIPSecIKEPort: 500
ceIPSecNATTPort: 4500
ceIPSecPSK: 0qZfLf2sx+bVlprOtS7jCuE1wjR9h/HnOfO326ReN63uTFY76bhUTThEqY+WjkLK
ceIPSecPSKSecret: submariner-ipsec-psk
clusterCIDR: ""
clusterID: site1
connectionHealthCheck:
enabled: true
intervalSeconds: 1
maxPacketLossCount: 5
debug: true
namespace: submariner-operator
natEnabled: true
repository: quay.io/submariner
serviceCIDR: ""
serviceDiscoveryEnabled: true
version: 0.12.0

    Site 2

    # oc get submariner submariner -n submariner-operator -o yaml
apiVersion: submariner.io/v1alpha1
kind: Submariner
metadata:
creationTimestamp: "2024-06-06T13:20:40Z"
finalizers:
- controllers.submariner.io/cleanup
generation: 1
managedFields:
- apiVersion: submariner.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
  f:spec:
    .: {}
    f:broker: {}
    f:brokerK8sApiServer: {}
    f:brokerK8sApiServerToken: {}
    f:brokerK8sCA: {}
    f:brokerK8sRemoteNamespace: {}
    f:brokerK8sSecret: {}
    f:ceIPSecDebug: {}
    f:ceIPSecIKEPort: {}
    f:ceIPSecNATTPort: {}
    f:ceIPSecPSK: {}
    f:ceIPSecPSKSecret: {}
    f:clusterCIDR: {}
    f:clusterID: {}
    f:connectionHealthCheck:
      .: {}
      f:enabled: {}
      f:intervalSeconds: {}
      f:maxPacketLossCount: {}
    f:debug: {}
    f:namespace: {}
    f:natEnabled: {}
    f:repository: {}
    f:serviceCIDR: {}
    f:serviceDiscoveryEnabled: {}
    f:version: {}
manager: subctl
operation: Update
time: "2024-06-06T13:20:40Z"
- apiVersion: submariner.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
  f:metadata:
    f:finalizers:
      .: {}
      v:"controllers.submariner.io/cleanup": {}
manager: submariner-operator
operation: Update
time: "2024-06-06T13:21:45Z"
- apiVersion: submariner.io/v1alpha1
fieldsType: FieldsV1
fieldsV1:
  f:status:
    .: {}
    f:clusterCIDR: {}
    f:clusterID: {}
    f:deploymentInfo: {}
    f:gatewayDaemonSetStatus:
      .: {}
      f:lastResourceVersion: {}
      f:mismatchedContainerImages: {}
      f:nonReadyContainerStates: {}
      f:status:
        .: {}
        f:currentNumberScheduled: {}
        f:desiredNumberScheduled: {}
        f:numberAvailable: {}
        f:numberMisscheduled: {}
        f:numberReady: {}
        f:observedGeneration: {}
        f:updatedNumberScheduled: {}
    f:gateways: {}
    f:globalnetDaemonSetStatus:
      .: {}
      f:mismatchedContainerImages: {}
    f:loadBalancerStatus: {}
    f:natEnabled: {}
    f:networkPlugin: {}
    f:routeAgentDaemonSetStatus:
      .: {}
      f:lastResourceVersion: {}
      f:mismatchedContainerImages: {}
      f:nonReadyContainerStates: {}
      f:status:
        .: {}
        f:currentNumberScheduled: {}
        f:desiredNumberScheduled: {}
        f:numberAvailable: {}
        f:numberMisscheduled: {}
        f:numberReady: {}
        f:observedGeneration: {}
        f:updatedNumberScheduled: {}
    f:serviceCIDR: {}
manager: submariner-operator
operation: Update
subresource: status
time: "2024-06-06T13:21:54Z"
name: submariner
namespace: submariner-operator
resourceVersion: "958559344"
uid: 43a6ff7a-b2d4-48c2-84a4-ad1f70ca8e86
spec:
broker: k8s
brokerK8sApiServer: api.ocp-psa-01.gbbper.priv:6443
brokerK8sApiServerToken: eyJhbGciOiJSUzI1NiIsImtpZCI6IngwR2pZY2NWTXJtRlI2T3pxVjFzSHU5OU5SNm9fUkt3V1BMTFJxcnV3LVkifQ.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJzdWJtYXJpbmVyLWs4cy1icm9rZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlY3JldC5uYW1lIjoiY2x1c3Rlci1zaXRlMi10b2tlbi0yNjRjcCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJjbHVzdGVyLXNpdGUyIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiZDM4OGM4ZjMtOGUwZS00NGE3LTlmMDUtMjNkMDFkODU3ZmE5Iiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50OnN1Ym1hcmluZXItazhzLWJyb2tlcjpjbHVzdGVyLXNpdGUyIn0.CP09gdfFSwOR0pk1XGA17Xbg67wOIWd4PqKtFhGUYk3mAvQEjkioM-Vm0UJDCAUqYjH6bzwnWkQxOT8odS8XpMQNd_hj0gHcGXjL5W8DAJzk01ritN-tNHcGJPWC_GxqPT5snAnyDWkL_KwwL4PKl3VEgJhhPbHNgp3fRBh-nbkqjeJ_2gfPr5BySCwAPHOIIZW1V4HTXqvhlFfIKLFiQtPD-PtcqV2P1uy02G7xMVjAMxcHhNFJXIDN8XNrjsCbf28NcoOs7WsjIb0iXI_bPhAQGvfWDWw7kRPIAi7z3OzIcGdrAsVQsw1HcBnAK2CN86b9NybuBGjtVzyUjd9O9QJSxT-OnzKxSj49K33emONX4de1H4hWK0biDccTv_x3gnyTi6HfTJOzfFObBaJVTbconNXfG13odmFiiHTY5z1h3LIfmNGVAi3DJcLBfRfWci2rAbp7niusUsQ0kWlQesIEWvhOG9XlPhp1Gp37R3pEchcQksBKDqUnFVs6o6XTG-Td92i4LdyGJeYslDMY5JHpouVzkrunOZhvwnTsROw6Tb6PbVien2UBGfuW_YGc_9BCgbB2-bxKlBa1NIS-tpr6gk3IjxNLxP5CssTG92888c1yM22USukcpm3nL7lpjCU9Y9Rb3DXynH9n5VjnJ19DpqfVwCU_Q3DrJfF2ilg
brokerK8sCA: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURNakNDQWhxZ0F3SUJBZ0lJZkRtandjUGF1K1F3RFFZSktvWklodmNOQVFFTEJRQXdOekVTTUJBR0ExVUUKQ3hNSmIzQmxibk5vYVdaME1TRXdId1lEVlFRREV4aHJkV0psTFdGd2FYTmxjblpsY2kxc1lpMXphV2R1WlhJdwpIaGNOTWpNd056SXdNVFV3TVRFMFdoY05Nek13TnpFM01UVXdNVEUwV2pBM01SSXdFQVlEVlFRTEV3bHZjR1Z1CmMyaHBablF4SVRBZkJnTlZCQU1UR0d0MVltVXRZWEJwYzJWeWRtVnlMV3hpTFhOcFoyNWxjakNDQVNJd0RRWUoKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTGNiT0dQbnRaTE1sR2xPQ1FHODFnWCt6dG1FMUFZWgp2RUkybFM4UCtCOGVkSDJCdk9GYkc2dWc1WUhSN1h4RkdlNVllSjZ4Tkw5M01OS05uK1BLcGhrNm5kdlJDTjZtCkF0RFI3WnBBczFYVGZEOHNHOHg3TWFYZTF5RU9SdmtZQXBSLzVOSFFhSnZzUWxqaXY3blQvUWNLTFRKbXpBUUsKS0ZwQ1pLd3NVc1R0NEtsYXVPLzlDWnZpMVZlcTV0NlJhTE1xdjZyREREUC8zcHo4U0NzTlYvR3hIeGhXVWl4dQpVK2RxbmF2RmxGa0xOdys0Qnh1SnFpcUJHNGNwV2p5cjFEMmJBcThvVnoxN1JaTmw2M09VdUxGa3VYdWlsUVBmCmkzSXN1WDQ2WTE0T01icHRzTTltNzZmdElSUWJNU0paZWxqdmlqVG9VKzZ2bFRNak0vUDdkQ0VDQXdFQUFhTkMKTUVBd0RnWURWUjBQQVFIL0JBUURBZ0trTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME9CQllFRkEwRwpyM2FvSk1UdVpmR3JjZlQrYm94NFZpQkdNQTBHQ1NxR1NJYjNEUUVCQ3dVQUE0SUJBUUE5dkRWRlpFOXhpUzRxCndaQS9DaFRjOG9IMDlCZmlBNTlpNjd0cldLMzZVVGo0SFZiM2ZMOEFXeXBnM2tZSGh2azZ2cUJnRHVmVFpOUjcKU2Y1OS9SY0w1VVRSdTA4bzlETHNvSTlFWUxkMjlRcTdjdHBsUmw1RmZ3UjY5ZjlOL2kwOFlRZDVKLzBLNHA1OAorR1M0WWhTR1d6c2oxUnZ2MUJBTkY2a056MFhGOG1lSVJVWkcyZFBBaFoyVWRxMnBPQ3d2MmFqZHZqWk14ZFBFCkdPMStSbDdBa3hNTEhKbXZDOGJVbEM2U3hsYm1BV2FiaTY5M3VkWjZUQkJwTjRwUkFNZnQvOHFzM3ZsRTlpMzcKajFTWitMTVNQR2xINWlwMUMzbjlDNlUxRlpBQU9vN09uYXJaSk9yZ09ua2h4MVBwM09wTDlYY1I3Q0Y2eE5sMQowbWtYcnY5aQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEUURDQ0FpaWdBd0lCQWdJSUVsSU52VkpFRVBVd0RRWUpLb1pJaHZjTkFRRUxCUUF3UGpFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNU2d3SmdZRFZRUURFeDlyZFdKbExXRndhWE5sY25abGNpMXNiMk5oYkdodmMzUXQKYzJsbmJtVnlNQjRYRFRJek1EY3lNREUxTURFeE5Gb1hEVE16TURjeE56RTFNREV4TkZvd1BqRVNNQkFHQTFVRQpDeE1KYjNCbGJuTm9hV1owTVNnd0pnWURWUVFERXg5cmRXSmxMV0Z3YVhObGNuWmxjaTFzYjJOaGJHaHZjM1F0CmMybG5ibVZ5TUlJQklqQU5CZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF2Qy82VVNrZEhkT3MKb2NuMFBLdm52RUtEcTZ3bzRkNXJMVGUybG5jam5UVWlRQUFtNkd5YkJRVC9BNTZyb3kxSUpPNGtpWXlscFFBRQpQaEk0dkIzMGdxNjZLS0czdlpkc3dpemt6OTFEVE9wV05ZY3Bmbzh4R3ZzZFZSM1RudVNnREdCVHV3NGNCYzV3Ci9LRXlnS3BtZk1WenJiS0cxbnlNY1daUHhBTkM4V3kyRm1NTSt1eVFFaVRCak9mVVBvWTZkZVVqUXpNZkp5dGsKR1NjMWordEUwejFsWkRZMjJQMW5yTDh3TkR3Y0R3T0gydGcvcHhVV2VkeEQ0SGFLMGwzRjQvOVlxYW56T21oRQo5Q2dRaEtHTGdERzUxUWE0WU1qWUl2bTJqUExvNUpOaU05SjlkZzhKSEdxNm5SM2hkZmI4TkhjY1B4bWRtRzhjClEwWVlYUWMwcXdJREFRQUJvMEl3UURBT0JnTlZIUThCQWY4RUJBTUNBcVF3RHdZRFZSMFRBUUgvQkFVd0F3RUIKL3pBZEJnTlZIUTRFRmdRVVNBZzgvY29iR1lUaitDb0xtMGdaaW9TUnljd3dEUVlKS29aSWh2Y05BUUVMQlFBRApnZ0VCQUtDYkVIN2lKQVN1ajRVcHo0ajd4RVBzY05RS0tOS3RzQ29TbHpuWTBuRTVYUjhwV3VVdmxvTS9xdVVQClUrL3RSWldTaXV3SFdEeUJHY2tubUJhOS92aE1HVzNNcjlCOWxlL0srVHpSQlg0aGF6YWRDbXU2b0p6K1lxWTcKRkQ4elVqYkxZN3BBQ1BFdkpJY3FRWTJTRDdlODd5N0Jnam5UM3lUMzdyd01pcXV6bHI1QmNjLzg3WjRCanNnTAo4MjAyUXVFYml6Q3ZpTnBqYUNyamw4YWtKTG1SRnZZVERiaTk0bzh1YklyVHZ6a1NpbmNhWnQ5eHlibWhySEt4CmhPWjd3Tnh4VzNQRUU5RVZRN1c3RVhLYUhadk1sVWZNUXVrSS84Y05uWDltdm9wSUY5MEVrb3hkd2JQZ00yZ3gKVXN5TllpRENkWENVcGFJWUJFNTUrbWQyb3pJPQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCi0tLS0tQkVHSU4gQ0VSVElGSUNBVEUtLS0tLQpNSUlEVERDQ0FqU2dBd0lCQWdJSVc4K0ZZTXhsdklJd0RRWUpLb1pJaHZjTkFRRUxCUUF3UkRFU01CQUdBMVVFCkN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyYVdObExXNWwKZEhkdmNtc3RjMmxuYm1WeU1CNFhEVEl6TURjeU1ERTFNREV4TkZvWERUTXpNRGN4TnpFMU1ERXhORm93UkRFUwpNQkFHQTFVRUN4TUpiM0JsYm5Ob2FXWjBNUzR3TEFZRFZRUURFeVZyZFdKbExXRndhWE5sY25abGNpMXpaWEoyCmFXTmxMVzVsZEhkdmNtc3RjMmxuYm1WeU1JSUJJakFOQmdrcWhraUc5dzBCQVFFRkFBT0NBUThBTUlJQkNnS0MKQVFFQXhLTWdMcmtyc2dSaE4zZEY1TFlBZForeFdmZXNJR0laa0FVZ25tV1Y3azdVbjJCUVkvbUNuekVPTHhibgpDNm56MWFKT3R1SDNoZWg4bE5MVGIxc1JQZkFpaVBIdVBOaDRrZXMxcDdiajVHcXd2MmRNc0tESUw2aHAxbE4xClJEWFNiZUw1ajd6UjFXbUJrL01taUQ2L2wwaWpIZGRjMXpRNmlFQ2p6MXFsbHl6UDM1ZW1VS1dwcG5iQld0S2IKYjJXNGlEdVhBQ0JvbWgzVTBkalZWYitYR3JiQkhaQ21iaWMrRWcvYi8ya3J2WFpZTG93N0QwTjVaRkRkZG84TgpRcFNabTVaOUZYNHE0LzJqMjVZUjAzSGNFd1RwNVI3UXFsWDYxWUZuT2dJWUNQNzB2d0N5aXpVclE4S3dnNGZ6ClZDaGl5Q1pCSXlHYzUrR3hzcUViZXVmL2VRSURBUUFCbzBJd1FEQU9CZ05WSFE4QkFmOEVCQU1DQXFRd0R3WUQKVlIwVEFRSC9CQVV3QXdFQi96QWRCZ05WSFE0RUZnUVVGamJyS3Y0NTFiWmgvaDM1SU9hREtuUnBpd013RFFZSgpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFKK2kzVURUN3Y1cVFpS1gzSW96Qk5MVzBiN3Fncy9tUjJueGFnOWo1RFFuCkZUelRGYjl6dDVUNG5VQ0hwdVRtdjZudWFvUWhrL1BUV2RDV21uSXp5WmVjMERxcCtWWXFCQnFyZGFKUU00b0oKVFdxQTU1WFZGU0s2aVpWS09wNDNETHg2aVhMN1NEWXNiQ000d1I2a3dSbEtib3Z3NjN4SE42QWtWcCtQRzk5dQo3RkR4REU5RG00SW1zd3hhcGFEMnFZTlhkdjRpS1R5cmtHWUg2elM1VmQwSUxtSkFaQjZxMk5sUnpEb2tSY3NTCkovRUJwU3pjVVFuUlRjbkkxM21VMTFqelZRVFRMYktsVXV2d01LU3ZPa0c5bXVsU1p3SUc0RnR5SVd5d1dGbjEKQ093ZUNlaTh5K0FpaE5BYkdET3U1Q0h6WGl3dDVxYko3T3hCeGY2RUh6bz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQotLS0tLUJFR0lOIENFUlRJRklDQVRFLS0tLS0KTUlJRGx6Q0NBbitnQXdJQkFnSUlNRmpCOEN4eWczWXdEUVlKS29aSWh2Y05BUUVMQlFBd1dURlhNRlVHQTFVRQpBd3hPYjNCbGJuTm9hV1owTFd0MVltVXRZWEJwYzJWeWRtVnlMVzl3WlhKaGRHOXlYMnh2WTJGc2FHOXpkQzF5ClpXTnZkbVZ5ZVMxelpYSjJhVzVuTFhOcFoyNWxja0F4TmpnNU9EWTNPRFV6TUI0WERUSXpNRGN5TURFMU5EUXgKTWxvWERUTXpNRGN4TnpFMU5EUXhNMW93V1RGWE1GVUdBMVVFQXd4T2IzQmxibk5vYVdaMExXdDFZbVV0WVhCcApjMlZ5ZG1WeUxXOXdaWEpoZEc5eVgyeHZZMkZzYUc5emRDMXlaV052ZG1WeWVTMXpaWEoyYVc1bkxYTnBaMjVsCmNrQXhOamc1T0RZM09EVXpNSUlCSWpBTkJna3Foa2lHOXcwQkFRRUZBQU9DQVE4QU1JSUJDZ0tDQVFFQTNCbTUKckxjdS9kcExrczZNcS90TGxvbjJTMFM2aXNzcEZSb1FRZjNwZXg2WDNHN0NLOWpQYldsQ0dWeElpKzdqTW9XYQpHYTYwc3hmeWZ6L0NDMlZpWnJBNUMrUU1GeTJoc29kalFxeEVDNWNVVERmaHRiVzVqYWltY25xUUgrQmRXeHhECmN5c0NWQVNaWFZ4WmRvaVhqTXAwb3A2QWkydzBZekE3UzJDSWFKaTBsTjNBamNFalFRWnNPTEJiMGNrQjFrWHkKNXJMWGI3cTR0WkxQVmtzVm94dncrOUFDNzdxdkNreGwzYWd6TEowZGRjYnFXUjEvS29OdHlRMklpb0pWVmtregoyUHB2ekJ1S3hYUWR0d2s5dWwzWWhwYWIvMEhxM2VjYlpyUGZmY2VVa3RZUjc2cW9RUzhBL0tQNFI1SG5RN3VZClJGUDdQbE9hS3BqdUNqVTVuUUlEQVFBQm8yTXdZVEFPQmdOVkhROEJBZjhFQkFNQ0FxUXdEd1lEVlIwVEFRSC8KQkFVd0F3RUIvekFkQmdOVkhRNEVGZ1FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0h3WURWUjBqQkJndwpGb0FVV3BIQmY2cndMRlpZVXVpYXdQU2VOSHZycDBzd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFLS2o4NW9MClU5a1JLdDlWRVdSNVp4UmdIYzc5TDkrckhYTldxVGpRV2xUcEhrd1B5RE5wLy83OHpUdmNLbkREaEZ1dG0yTE8KMHZCbnlBNlZ3QmdWQS9hR1BnUENGUkxRZk9ndEtHNVFhTW8zUC9ZR3RyN0NHb1NpYUFPblRTZjRybjFlVy9oUAovOGlOZEFzZzVJYU5IMFkrcGM1dEVkUDhUOUNpQmUwUDViNGk3ZFNidXVBVFJOK2U0b2ZvRjV5dEd3RHJ5RjZqCm9MSi9ZUHZFSEtWanhxUWpCYkltdzVVMHJMSWpSZEQ3dlMybmNEWm9YY3Q1djBzeUxuN3JabWd0aWJJRkpFYTAKVng5c2RETkZGSWxWcHBYVXU4RnJ2RHY4TUdsNk9CQUVSZXBCQjhndVpsS0R5TlRNaU1DM3hRTjB0N2wxaXNGagprSWN2ZFFuL1dnV3I2cmc9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0KLS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUVtVENDQTRHZ0F3SUJBZ0lDQzI0d0RRWUpLb1pJaHZjTkFRRUxCUUF3Z2JjeEN6QUpCZ05WQkFZVEFrbFUKTVE4d0RRWURWUVFJRXdaTmIyUmxibUV4RnpBVkJnTlZCQWNURGtObGJuUnlieUJ6WlhKMmFYcHBNUlF3RWdZRApWUVFLRXd0SGNuVndjRzhnUWxCRlVqRVdNQlFHQTFVRUN4TU5RbEJGVWlCVFpYSjJhV05sY3pFVk1CTUdBMVVFCkRCTU1ZMkVnWW5CbGNpQXlNRE00TVRrd053WURWUVFERXpCSGNuVndjRzhnUWxCRlVpQXRJRk5sY25acGVtbHYKSUdScElFTmxjblJwWm1sallYcHBiMjVsSUVsdWRHVnlibUV3SGhjTk1qTXdNakkyTWpNd01EQXdXaGNOTWpVdwpNakkzTWpJMU9UVTVXakNCcFRFTE1Ba0dBMVVFQmhNQ1NWUXhDekFKQmdOVkJBZ01BazFQTVE4d0RRWURWUVFICkRBWk5iMlJsYm1FeEdUQVhCZ05WQkFvTUVFSlFSVklnUW1GdVkyRWdVeTV3TGtFeE5UQXpCZ05WQkFzTUxGVm0KWmk0Z1UybHpkR1Z0YVNCRWFYQmhjblJwYldWdWRHRnNhU0JsSUVOdmJXMTFibWxqWVhScGIyNXpNU1l3SkFZRApWUVFEREIwcUxtRndjSE11YjJOd0xYQnpZUzB3TVM1blltSndaWEl1Y0hKcGRqQ0NBU0l3RFFZSktvWklodmNOCkFRRUJCUUFEZ2dFUEFEQ0NBUW9DZ2dFQkFLdm1MakhlaUx3L2p0d3hUUFcvUTJzSXdjb3RWSDVEcWxGOEtvMjEKUGExWml6T0dFTXk5R0hhNUEvYXZ6clFXdW10RGNVTExqemdRaDB6N29hWDNxSWJMbklRMHllbDU5Y1ZGZ1ZWTwpVSFpYN29kaUlPMzdkNVhuZ014TnBlcG1jWFFsOEEzV3hkVkRxRnlOT295OGRVT1dCWk1ONGhmLzN3ZUVhWVczCis5S0FlY2JLbS8wUEcrdkFJNms1bm1sUmtYK2w4T0tIVit6US9FMkd3bjByT1JHTml2YjFOV0lYYzZpVENqRjQKL1hmb1c2aE9sTzJWNWpnM0cvSlJ6VUg5UFUwRjJML3dielNmUU15MHF4SzZwOUZGNHdPL3ZqNlNuMGN6NlFzdwpYeWI1Rm01NXpIYitONE04cVFsRU9pc01iYmZhME9va3NCaGVCU05reC9sRWg3VUNBd0VBQWFPQnZqQ0J1ekEvCkJnbGdoa2dCaHZoQ0FRMEVNaFl3UjJWdVpYSmhkR1ZrSUdKNUlIUm9aU0JUWldOMWNtbDBlU0JUWlhKMlpYSWcKWm05eUlIb3ZUMU1nS0ZKQlEwWXBNQ2dHQTFVZEVRUWhNQitDSFNvdVlYQndjeTV2WTNBdGNITmhMVEF4TG1kaQpZbkJsY2k1d2NtbDJNQTRHQTFVZER3RUIvd1FFQXdJRXNEQWRCZ05WSFE0RUZnUVVZVGh5ajVSbVhud1NmdEhKCkE3Y3VJQnEwT0Zvd0h3WURWUjBqQkJnd0ZvQVVxV29Oc1FyVXhpLzI5TERDN1JPNkpPTFpnUEF3RFFZSktvWkkKaHZjTkFRRUxCUUFEZ2dFQkFBRmlJUHZGdHJYaFM0UExpNS94bmZkV1k1UDNnZERTZlJMMDRoWjROd2JZSTVXZwpZRklMVnlwTnVLNVNSTk1hSWw3WkFaUytnUkJoQVRCVStIdjNLN083VHlLaC9UQ2tnOVpHZzI2SVhxc0t5UHZ0CmFPSUx5SDFTZkRiYXhPN3RPclhrTTA3eGd6S0hocUhWWlQ5YXpPUnVrVlJaa01nd0gySGpmdWNTV0ZrTis2eDIKdm1HempBQWlKQ1JGY0JwNnRLYTY0b3RGdFRmY2FNYVZVRlllZkRqZ285b2tnWTRqZ2UwWXlXQkErc1hOUWsydgpVWHhrMm1IYitjSlAxS1I5ayswOEt1YXFKYkJCS0s0RC9tWHkwcCtYODdDYWpZdFRxQ0lHSXk4T3JROTFIYUxiClhYQzlSMWRYV2krMkdrTmI0SkVhKyt0aFVCaVNsQldycHpycURPUT0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
brokerK8sRemoteNamespace: submariner-k8s-broker
brokerK8sSecret: broker-secret-2brrr
ceIPSecDebug: false
ceIPSecIKEPort: 500
ceIPSecNATTPort: 4500
ceIPSecPSK: 0qZfLf2sx+bVlprOtS7jCuE1wjR9h/HnOfO326ReN63uTFY76bhUTThEqY+WjkLK
ceIPSecPSKSecret: submariner-ipsec-psk
clusterCIDR: ""
clusterID: site2
connectionHealthCheck:
enabled: true
intervalSeconds: 1
maxPacketLossCount: 5
debug: true
namespace: submariner-operator
natEnabled: true
repository: quay.io/submariner
serviceCIDR: ""
serviceDiscoveryEnabled: true
version: 0.12.0
status:
clusterCIDR: 10.132.0.0/14
clusterID: site2
deploymentInfo: {}
gatewayDaemonSetStatus:
lastResourceVersion: "951307195"
mismatchedContainerImages: false
nonReadyContainerStates: []
status:
  currentNumberScheduled: 3
  desiredNumberScheduled: 3
  numberAvailable: 3
  numberMisscheduled: 0
  numberReady: 3
  observedGeneration: 1
  updatedNumberScheduled: 3
gateways:
- connections:
- endpoint:
    backend: libreswan
    backend_config:
      natt-discovery-port: "4490"
      preferred-server: "false"
      public-ip: dns:control-1-ru4.ocp-psa-01.gbbper.priv
      udp-port: "4500"
    cable_name: submariner-cable-site1-192-168-54-32
    cluster_id: site1
    healthCheckIP: 10.129.0.2
    hostname: control-1-ru4.ocp-psa-01.gbbper.priv
    nat_enabled: true
    private_ip: 192.168.54.32
    public_ip: 192.168.54.32
    subnets:
    - 172.30.0.0/16
    - 10.128.0.0/14
  latencyRTT:
    average: 1.181959ms
    last: 1.103177ms
    max: 66.129596ms
    min: 924.003µs
    stdDev: 704.858µs
  status: connected
  statusMessage: ""
  usingIP: 192.168.54.32
haStatus: active
localEndpoint:
  backend: libreswan
  backend_config:
    natt-discovery-port: "4490"
    preferred-server: "false"
    public-ip: dns:control-1-ru2.ocp-psb-01.gbbper.priv
    udp-port: "4500"
  cable_name: submariner-cable-site2-192-168-126-30
  cluster_id: site2
  healthCheckIP: 10.133.0.2
  hostname: control-1-ru2.ocp-psb-01.gbbper.priv
  nat_enabled: true
  private_ip: 192.168.126.30
  public_ip: 192.168.126.30
  subnets:
  - 172.31.0.0/16
  - 10.132.0.0/14
statusFailure: ""
version: release-0.12-621cbce
- connections: []
haStatus: passive
localEndpoint:
  backend: libreswan
  backend_config:
    natt-discovery-port: "4490"
    preferred-server: "false"
    public-ip: dns:control-1-ru3.ocp-psb-01.gbbper.priv
    udp-port: "4500"
  cable_name: submariner-cable-site2-192-168-126-31
  cluster_id: site2
  healthCheckIP: 10.132.0.2
  hostname: control-1-ru3.ocp-psb-01.gbbper.priv
  nat_enabled: true
  private_ip: 192.168.126.31
  public_ip: 192.168.126.31
  subnets:
  - 172.31.0.0/16
  - 10.132.0.0/14
statusFailure: ""
version: release-0.12-621cbce
- connections: []
haStatus: passive
localEndpoint:
  backend: libreswan
  backend_config:
    natt-discovery-port: "4490"
    preferred-server: "false"
    public-ip: dns:control-1-ru4.ocp-psb-01.gbbper.priv
    udp-port: "4500"
  cable_name: submariner-cable-site2-192-168-126-32
  cluster_id: site2
  healthCheckIP: 10.134.0.2
  hostname: control-1-ru4.ocp-psb-01.gbbper.priv
  nat_enabled: true
  private_ip: 192.168.126.32
  public_ip: 192.168.126.32
  subnets:
  - 172.31.0.0/16
  - 10.132.0.0/14
statusFailure: ""
version: release-0.12-621cbce
globalnetDaemonSetStatus:
mismatchedContainerImages: false
loadBalancerStatus: {}
natEnabled: true
networkPlugin: OVNKubernetes
routeAgentDaemonSetStatus:
lastResourceVersion: "945546125"
mismatchedContainerImages: false
nonReadyContainerStates: []
status:
  currentNumberScheduled: 12
  desiredNumberScheduled: 12
  numberAvailable: 12
  numberMisscheduled: 0
  numberReady: 12
  observedGeneration: 1
  updatedNumberScheduled: 12
serviceCIDR: 172.31.0.0/16
  5. oc get network cluster -o yaml Site 1 
    # oc get network cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: "2023-07-20T15:08:25Z"
generation: 2
managedFields:
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
  f:spec:
    .: {}
    f:clusterNetwork: {}
    f:externalIP:
      .: {}
      f:policy: {}
    f:networkType: {}
    f:serviceNetwork: {}
  f:status: {}
manager: cluster-bootstrap
operation: Update
time: "2023-07-20T15:08:25Z"
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
  f:status:
    f:clusterNetwork: {}
    f:clusterNetworkMTU: {}
    f:networkType: {}
    f:serviceNetwork: {}
manager: cluster-network-operator
operation: Update
time: "2023-07-20T15:40:50Z"
name: cluster
resourceVersion: "6285"
uid: 42ed0cab-58a1-4229-8d8a-78ddf7430518
spec:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
externalIP:
policy: {}
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16
status:
clusterNetwork:
- cidr: 10.128.0.0/14
hostPrefix: 23
clusterNetworkMTU: 1400
networkType: OVNKubernetes
serviceNetwork:
- 172.30.0.0/16

    Site 2

    #  oc get network cluster -o yaml
apiVersion: config.openshift.io/v1
kind: Network
metadata:
creationTimestamp: "2023-08-16T08:57:56Z"
generation: 2
managedFields:
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
  f:spec:
    .: {}
    f:clusterNetwork: {}
    f:externalIP:
      .: {}
      f:policy: {}
    f:networkType: {}
    f:serviceNetwork: {}
  f:status: {}
manager: cluster-bootstrap
operation: Update
time: "2023-08-16T08:57:56Z"
- apiVersion: config.openshift.io/v1
fieldsType: FieldsV1
fieldsV1:
  f:status:
    f:clusterNetwork: {}
    f:clusterNetworkMTU: {}
    f:networkType: {}
    f:serviceNetwork: {}
manager: cluster-network-operator
operation: Update
time: "2023-08-16T09:31:10Z"
name: cluster
resourceVersion: "5877"
uid: 7b575eb7-3655-4244-a157-71a87077b829
spec:
clusterNetwork:
- cidr: 10.132.0.0/14
hostPrefix: 23
externalIP:
policy: {}
networkType: OVNKubernetes
serviceNetwork:
- 172.31.0.0/16
status:
clusterNetwork:
- cidr: 10.132.0.0/14
hostPrefix: 23
clusterNetworkMTU: 1400
networkType: OVNKubernetes
serviceNetwork:
- 172.31.0.0/16
BhavaniYalamanchili commented 2 weeks ago

@dfarrell07 If the submariner needs to be upgraded, Can you point out which submariner version is compatible with OCP 4.10?

yboaron commented 2 weeks ago

@BhavaniYalamanchili

Here are the outputs:

oc get pods -A -1 name=ovnkube-db No resources found

Did you mean oc get pods -A -l name=ovnkube-db ?

BhavaniYalamanchili commented 2 weeks ago

@yboaron Sorry typo, it is -l not -1

Did you mean oc get pods -A -l name=ovnkube-db ?

Yes

tpantelis commented 2 weeks ago

@dfarrell07 If the submariner needs to be upgraded, Can you point out which submariner version is compatible with OCP 4.10?

You had created https://github.com/submariner-io/submariner/issues/2955 a while back where you were using Submariner 0.16.x and OCP 4.15 so I'm curious why you would now be using much older versions of each (which are no longer supported)....

yboaron commented 2 weeks ago

A.

oc get pods -A -1 name=ovnkube-db No resources found

Hmm, that's weird, b/c if no pod with ovnkube-db label is found Submariner shouldn't recognize CNI plugin as OVN-K8S, and OVN-K8S was detected successfully for site2

B. What version of subctl do you use ?

C. Also think it is better to upgrade Submariner, for OCP 4.10 you can upgrade to Submariner 0.14.

D. If you decide to stay with version 0.12, and still hit this issue please upload subctl gather from both clusters.

BhavaniYalamanchili commented 2 weeks ago

@tpantelis This is a different cluster setup

@yboaron B. 0.12.0 subctl only is being used

C. Any version of 0.14.x would work on OCP 4.10?

D. Sure, will try to get the subctl gather from both clusters, it might take some time. Meanwhile, I have found some error log statements in submariner-operator logs of Site 1, Please check this out.

[90m2024-06-06T13:24:47.742Z[0m [32mINF[0m ..e-arguments/main.go:174 cmd                  Could not generate and serve custom resource metrics [36merror=[0m[31m"error initializing metrics: discovering resource information failed for Submariner in submariner.io/v1alpha1: unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m

[90m2024-06-06T13:25:22.495Z[0m [32mINF[0m ..e-arguments/main.go:229 cmd                  Could not create ServiceMonitor object [36merror=[0m[31m"unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m

[90m2024-06-06T13:25:40.455Z[0m [1m[31mERR[0m[0m ..oller/controller.go:267 ..mariner-controller Reconciler error [36merror=[0m[31m"unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request"[0m [36mname=[0msubmariner [36mnamespace=[0msubmariner-operator [36mreconciler group=[0msubmariner.io [36mreconciler kind=[0mSubmariner
yboaron commented 1 week ago

C. Any version of 0.14.x would work on OCP 4.10?

Yep.

BhavaniYalamanchili commented 1 week ago

@yboaron There is a concern about sharing all the logs. Could you let me know if you want any specific logs to look for the exact error?

yboaron commented 1 week ago

Please share the output of oc get submariner submariner -n submariner-operator -o yaml , from site1 , couldn't see the status section in the ^^ you attached above. and also submariner-operator pod logs from both clusters

BhavaniYalamanchili commented 1 week ago

@yboaron

The output of the command oc get submariner submariner -n submariner-operator -o yaml is displayed till the spec only the status section is not displayed, I wonder why In the yaml file that is collected also its the same

Here are the submariner-operator pod logs you asked Please change the the extension to zip when downloaded SIte1 logs.txt SIte2 logs.txt

yboaron commented 1 week ago

According to the site1 logs, submariner-operator failed to reconcile due to [1] error, and therefore does not update submariner.status section.

Please address this issue and let us know how it goes.

[1]

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

tpantelis commented 1 week ago

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

This looks like an incompatibility with the K8s version your using, ie the K8s version is newer than what's supported by Submariner 0.12. As mentioned earlier, 0.12 is no longer maintained so I strongly suggest upgrading Submariner.

BhavaniYalamanchili commented 1 week ago

In the past when we had an issue with Submariner 0.12.0 and OCP 4.10, then it was said by your team that the OCP version must be 4.11+ for the submariner 0.13. https://github.com/submariner-io/submariner/issues/1978#issuecomment-1227083099 Also pointed to the third bullet of the doc https://github.com/submariner-io/releases/releases/tag/v0.13.0 So will the submariner 0.14.6 work on OCP 4.10?

Also in the same ticket, it was mentioned that the Submariner does not support OVN on OCP 4.9 and 4.10 https://github.com/submariner-io/submariner/issues/1978#issuecomment-1227101915

tpantelis commented 1 week ago

You're better off upgrading both Submariner and OCP to supported versions.

BhavaniYalamanchili commented 1 week ago

@yboaron @tpantelis

We upgraded the Submariner to 0.14.6 version, and even after the upgrade its the same error message we are seeing and some add on errors

Site 1

./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-1-kubeconfig
I0620 11:47:51.523338 1183225 request.go:601] Waited for 1.010687181s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/ibmcpcs.ibm.com/v1?timeout=32s
I0620 11:48:01.523343 1183225 request.go:601] Waited for 10.921594654s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/machineconfiguration.openshift.io/v1?timeout=32s
Cluster "site1"
I0620 11:48:11.542570 1183225 request.go:601] Waited for 5.340290374s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/satellite.isf.ibm.com/v1?timeout=32s
 ✓ Detecting broker(s)
NAMESPACE               NAME                COMPONENTS                        GLOBALNET   GLOBALNET CIDR   DEFAULT GLOBALNET SIZE   DEFAULT DOMAINS
submariner-k8s-broker   submariner-broker   service-discovery, connectivity   no          242.0.0.0/8      65536

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP        NAT   CABLE DRIVER   SUBNETS                        STATUS   RTT avg.
control-1-ru4.ocp-psb-01.gbbpe   site2     192.168.126.32   no    libreswan      172.31.0.0/16, 10.132.0.0/14   error    0s

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP      PUBLIC IP        CABLE DRIVER   TYPE
site1     192.168.54.30    192.168.54.30    libreswan      local
site2     192.168.126.32   192.168.126.32   libreswan      remote
site1     192.168.54.31    192.168.54.31    libreswan      local
site1     192.168.54.32    192.168.54.32    libreswan      local

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.ocp-psa-01.gbbpe   active      0 connections out of 1 are established
control-1-ru3.ocp-psa-01.gbbpe   passive     There are no connections
control-1-ru4.ocp-psa-01.gbbpe   passive     There are no connections

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:
        Service CIDRs:   []
        Cluster CIDRs:   []

 ✓ Showing versions
COMPONENT             REPOSITORY           VERSION
submariner-gateway    quay.io/submariner   0.14.6
submariner-operator   quay.io/submariner   0.14.6
sh-4.4$ ./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-1-kubeconfig [root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc rsh isf-metrodr-operator-controller-manager-c88b64864-5m9bb
error: You must be logged in to the server (Unauthorized)
[root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc login https://api.ocp-psb-01.gbbper.priv:6443/ -u kubeadmin -p 43aTz-Ipzat-AzVqt-4YhpF --insecure-skip-tls-verify=true
Login successful.

You have access to 76 projects, the list has been suppressed. You can list all projects with ' projects'

Using project "ibm-spectrum-fusion-ns".
[root@bstcp4s-p-sl-01 subctl-v0.14.6]# oc rsh isf-metrodr-operator-controller-manager-c88b64864-5m9bb                                                        Defaulting container name to manager.
Use 'oc describe pod/isf-metrodr-operator-controller-manager-c88b64864-5m9bb -n ibm-spectrum-fusion-ns' to see all of the containers in this pod.
sh-4.4$ cd /tmp/subctl_new/
sh-4.4$ ./subctl-v0.14.6-linux-amd64 diagnose all --kubeconfig=/tmp/site-1-kubeconfig
Cluster "site1"
I0620 11:49:27.218182 1183270 request.go:601] Waited for 1.008715728s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/spp-data-protection.isf.ibm.com/v1alpha1?timeout=32s
I0620 11:49:37.417620 1183270 request.go:601] Waited for 11.208065595s due to client-side throttling, not priority and fairness, request: GET:https://api.ocp-psa-01.gbbper.priv:6443/apis/authorization.openshift.io/v1?timeout=32s
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✗ Checking Submariner support for the CNI network plugin
 ✗ The detected CNI plugin ("") is not supported by Submariner. Supported plugins: [generic canal-flannel weave-net OpenShiftSDN OVNKubernetes calico kindnet]
 ✗ Checking gateway connections
 ✗ Connection to cluster "site2" is not established. Connection details:
{
  "status": "error",
  "statusMessage": "Failed to successfully ping the remote endpoint IP \"10.134.0.2\"",
  "endpoint": {
    "cluster_id": "site2",
    "cable_name": "submariner-cable-site2-192-168-126-32",
    "healthCheckIP": "10.134.0.2",
    "hostname": "control-1-ru4.ocp-psb-01.gbbper.priv",
    "subnets": [
      "172.31.0.0/16",
      "10.132.0.0/14"
    ],
    "private_ip": "192.168.126.32",
    "public_ip": "192.168.126.32",
    "nat_enabled": true,
    "backend": "libreswan",
    "backend_config": {
      "natt-discovery-port": "4490",
      "preferred-server": "false",
      "public-ip": "dns:control-1-ru4.ocp-psb-01.gbbper.priv",
      "udp-port": "4500"
    }
  },
  "usingIP": "192.168.126.32",
  "latencyRTT": {
    "last": "0s",
    "min": "0s",
    "average": "0s",
    "max": "0s",
    "stdDev": "0s"
  }
}
 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs
 ✗ Checking Submariner pods
 ✗ Error obtaining Daemonset "submariner-routeagent": daemonsets.apps "submariner-routeagent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-agent": deployments.apps "submariner-lighthouse-agent" not found
 ✗ Error obtaining Deployment "submariner-lighthouse-coredns": deployments.apps "submariner-lighthouse-coredns" not found
 ✗ Error obtaining Daemonset "submariner-metrics-proxy": daemonsets.apps "submariner-metrics-proxy" not found
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported
 ✗ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✗ The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 172.31.0.0. Please check that your firewall configuration allows UDP/4800 traffic.
 ✓ Globalnet is not installed - skipping

 ⚠ Service discovery is not installed

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.14.6

Site 2

sh-4.4$ ./subctl-v0.14.6-linux-amd64 show all --kubeconfig=/tmp/site-2-kubeconfig
Cluster "local-config"
 ✓ Detecting broker(s)
 ✓ No brokers found

 ✓ Showing Connections
GATEWAY                          CLUSTER   REMOTE IP       NAT   CABLE DRIVER   SUBNETS                        STATUS      RTT avg.
control-1-ru2.ocp-psa-01.gbbpe   site1     192.168.54.30   no    libreswan      172.30.0.0/16, 10.128.0.0/14   connected   1.400162ms

 ✓ Showing Endpoints
CLUSTER   ENDPOINT IP      PUBLIC IP        CABLE DRIVER   TYPE
site2     192.168.126.30   192.168.126.30   libreswan      local
site2     192.168.126.31   192.168.126.31   libreswan      local
site2     192.168.126.32   192.168.126.32   libreswan      local
site1     192.168.54.30    192.168.54.30    libreswan      remote

 ✓ Showing Gateways
NODE                             HA STATUS   SUMMARY
control-1-ru2.ocp-psb-01.gbbpe   passive     There are no connections
control-1-ru3.ocp-psb-01.gbbpe   passive     There are no connections
control-1-ru4.ocp-psb-01.gbbpe   active      All connections (1) are established

 ✓ Showing Network details
    Discovered network details via Submariner:
        Network plugin:  OVNKubernetes
        Service CIDRs:   [172.31.0.0/16]
        Cluster CIDRs:   [10.132.0.0/14]

 ✓ Showing versions
COMPONENT                       REPOSITORY           VERSION
submariner-gateway              quay.io/submariner   0.14.6
submariner-routeagent           quay.io/submariner   0.14.6
submariner-operator             quay.io/submariner   0.14.6
submariner-lighthouse-agent     quay.io/submariner   0.14.6
submariner-lighthouse-coredns   quay.io/submariner   0.14.6

On site2 there is a difference from previous version output

sh-4.4$ /tmp/subctl_new/subctl-v0.14.6-linux-amd64 diagnose all --kubeconfig=/tmp/site-2-kubeconfig
Cluster "local-config"
 ✓ Checking Submariner support for the Kubernetes version
 ✓ Kubernetes version "v1.23.12+8a6bfe4" is supported

 ✓ Checking Submariner support for the CNI network plugin
 ✓ The detected CNI network plugin ("OVNKubernetes") is supported
 ✗ Checking OVN version
 ✗ The ovn-nb database version 5.35.1 is less than the minimum supported version 6.1.0
 ✓ Checking gateway connections
 ✓ All connections are established
 ✓ Non-Globalnet deployment detected - checking if cluster CIDRs overlap
 ✓ Clusters do not have overlapping CIDRs
 ✗ Checking Submariner pods
 ✗ The desired number of replicas for Deployment "submariner-networkplugin-syncer" (1) does not match the actual number running (0)
 ⚠ Pod "submariner-networkplugin-syncer-547fff98b6-qbz8v" has restarted 13 times
 ✓ Checking Submariner support for the kube-proxy mode
 ✓ The kube-proxy mode is supported
 ✓ Checking the firewall configuration to determine if intra-cluster VXLAN traffic is allowed
 ✓ This check is not necessary for the OVNKubernetes CNI plugin
 ✓ The firewall configuration allows intra-cluster VXLAN traffic
 ✓ Globalnet is not installed - skipping

 ✓ Checking if services have been exported properly
 ✓ All services have been exported properly

Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually.

subctl version: v0.14.6
yboaron commented 6 days ago

First of all sorry I wasn't clear in my previous answer, Submariner 0.14.x can work with OCP 4.10 but with CNI other than OVN-K.

As suggested, please upgrade also OCP to version 4.11+, as OVN-K requires OVN NorthBound DB version 6.1.0+, available with OCP 4.11.0+

BhavaniYalamanchili commented 5 days ago

@yboaron we have some concerns about updating the OCP 4.10 to 4.11, we want to get the submariner issue fixed through which a component that is dependent on the submariner will be fixed and then only we can proceed to the OCP upgrade.

@tpantelis said that it looks like an incompatibility issue

This looks like an incompatibility with the K8s version your using, ie the K8s version is newer than what's supported by Submariner 0.12.

So, which K8s version is compatible with Submariner 0.12.0?

Another point I see is that the Submariner 0.12. and also 0.14. versions are trying to fetch a pod that has the label name=ovnkube-db on it

And we don't have any pods like that on both sites, 

# oc get pods -A -l name=ovnkube-db

# oc get pods -n openshift-ovn-kubernetes -l name=ovnkube-db
No resources found in openshift-ovn-kubernetes namespace.

But the Site 2 is able to detect the plugin Is there any other way that Site2 is able to detect the CNI plugin?

Here I want to mention a point on what happened before this issue There was an issue with Site2, it wasn't accessible neither via oc cli nor via web-console) when we approached RedHat, the RedHat support team recreated the OVN database on Site2 and then we observed this issue with the submariner. Do you think the recreation of the OVN database on Site 2 is in any way relatable to this issue?

yboaron commented 3 days ago

Well, I'm afraid it's a chicken and egg problem here, Submariner doesn't support OCP 4.10 with OVN-K as cni and you want to upgrade OCP after Submariner issue resolved.

I think the best thing would be to uninstall Submariner, upgrade OCP, reinstall Submariner (of course upgrading submariner version).

Maybe you can try resolve Submariner issue before upgrading OCP using the following workaround:

[1]

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server is currently unable to handle the request

[2] https://cloud.ibm.com/docs/containers?topic=containers-debug_metrics_server https://pet2cattle.com/2021/05/unable-to-retrieve-the-complete-list-of-server-apis