yboaron
commented
2 weeks ago Thanks for reaching out @aswinayyolath.
A. As mentioned in Slack discussion, inter-cluster libreswan tunnel is up and communication between gw nodes is fine while communication from non-GW node to gw node is failing.
further dapath investigation is needed here, I assume that for some reason (maybe infra firewall, connection tracking) ingress packet is being dropped in gwnode@clusterX to nongwnode@clusterX segment.
Can you please run ping from non-gw node@sub1 to gw-node@sub2 (for gw-node@sub2 IP address you should use endpoint healthcheck IP == 242.0.255.254) and tcpdump the gw node and non-gw node on cluster sub1 ?
B. Also, this is not relevant to datapath issue, but I noticed that Submariner detected the CNI as generic instead of flannel, Submariner uses this code to discover network details for flannel CNI. Can you share please the daemonsets list from kube-system namespace ? and if one of those daemonset's name contains the string 'flannel' , share also the volume list of this ds (and if exists volume/configmap with name containing 'flannel' substring share its content)
aswinayyolath
What happened: I deployed Submariner with GlobalNet across two Kubernetes clusters. DNS resolution works as expected, but connectivity to services across clusters fails with a
Host is unreachableerror.More info is available in below link
https://kubernetes.slack.com/archives/C010RJV694M/p1730390376380879
What you expected to happen:
curlrequests from a pod in cluster2 to a service exposed via Submariner in cluster1 should succeed, indicating that cross-cluster communication is functioning.How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?:
Environment:
subctl diagnose all):Cluster 1 info
Cluster 2 info
``` Aswin 🔥🔥🔥 $ subctl diagnose all --kubeconfig /Users/aswina/Downloads/sub2 Cluster "sub2" ✓ Checking Submariner support for the Kubernetes version ✓ Kubernetes version "v1.30.6" is supported ✗ Globalnet deployment detected - checking that globalnet CIDRs do not overlap ✗ Error getting the Broker's REST config: error getting auth rest config: Get "https://9.66.245.122:6443/apis/submariner.io/v1/namespaces/submariner-k8s-broker/clusters/any": tls: failed to verify certificate: x509: "kube-apiserver" certificate is not trusted ⚠ Checking Submariner support for the CNI network plugin ⚠ Submariner could not detect the CNI network plugin and is using ("generic") plugin. It may or may not work. ✓ Checking gateway connections ✗ Checking route agent connections ✗ Connection to cluster "cluster1" is not established. Connection details: { "status": "error", "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"", "spec": { "cluster_id": "cluster1", "cable_name": "submariner-cable-cluster1-10-21-101-75", "healthCheckIP": "242.0.255.254", "hostname": "sub1-worker-1.fyre.ibm.com", "subnets": [ "242.0.0.0/16" ], "private_ip": "10.21.101.75", "public_ip": "129.41.87.4", "nat_enabled": true, "backend": "libreswan", "backend_config": { "natt-discovery-port": "4490", "preferred-server": "false", "udp-port": "4500" } } } ✗ Connection to cluster "cluster1" is not established. Connection details: { "status": "error", "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"", "spec": { "cluster_id": "cluster1", "cable_name": "submariner-cable-cluster1-10-21-101-75", "healthCheckIP": "242.0.255.254", "hostname": "sub1-worker-1.fyre.ibm.com", "subnets": [ "242.0.0.0/16" ], "private_ip": "10.21.101.75", "public_ip": "129.41.87.4", "nat_enabled": true, "backend": "libreswan", "backend_config": { "natt-discovery-port": "4490", "preferred-server": "false", "udp-port": "4500" } } } ✗ Connection to cluster "cluster1" is not established. Connection details: { "status": "error", "statusMessage": "Failed to successfully ping the remote endpoint IP \"242.0.255.254\"", "spec": { "cluster_id": "cluster1", "cable_name": "submariner-cable-cluster1-10-21-101-75", "healthCheckIP": "242.0.255.254", "hostname": "sub1-worker-1.fyre.ibm.com", "subnets": [ "242.0.0.0/16" ], "private_ip": "10.21.101.75", "public_ip": "129.41.87.4", "nat_enabled": true, "backend": "libreswan", "backend_config": { "natt-discovery-port": "4490", "preferred-server": "false", "udp-port": "4500" } } } ✓ Checking Submariner support for the kube-proxy mode ✓ The kube-proxy mode is supported ✗ Checking that firewall configuration allows intra-cluster VXLAN traffic ✗ The tcpdump output from the sniffer pod does not contain the expected remote endpoint IP 242.0.0.0. Please check that your firewall configuration allows UDP/4800 traffic. Actual pod output: tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on vx-submariner, link-type EN10MB (Ethernet), snapshot length 262144 bytes 0 packets captured 0 packets received by filter 0 packets dropped by kernel ✓ Checking that Globalnet is correctly configured and functioning ✓ Checking that services have been exported properly Skipping inter-cluster firewall check as it requires two kubeconfigs. Please run "subctl diagnose firewall inter-cluster" command manually. subctl version: v0.19.0 ```
subctl gather):sub1.zip
sub2.zip
K8S is installed on ubuntu VM
OS INFO