pod to pod connectivity not working after calico v3.28 deployment in K8s 1.28v

udhayd commented 1 month ago

Expected Behavior

To have pod to pod communication NA ## Current Behavior We have upgraded CNI Plugin from canal v3.23 to calico v3.28 in k8s v1.28 cluster, after deployment of calico v3.28 plugin pod to pod communication didnt work. Pods were not able to reach coredns getting i/o timeout. Calico kube-controllers & calico-nodes are up & running fine. However , we are facing this issue. **Pod Error to coredns:** communications error to 172.27.3.207#53: timed out NA ## Possible Solution

Steps to Reproduce (for bugs)

1. 2. 3. 4.

Context

Due to this issue , we were not able to access the workloads.

Your Environment

Calico version v3.28.0
Orchestrator version (e.g. kubernetes, mesos, rkt): kubernetes v1.28
Operating System and version: RHEL 7.9 & RHEL 8.9 , RHEL 8.10
Link to your project (optional):

IP Pool:

apiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: annotations: projectcalico.org/metadata: '{"uid":"68c046a0-6f6a-4fd4-9207-b47a98b40e4a","creationTimestamp":"2022-11-08T09:20:53Z"}' creationTimestamp: "2022-11-08T09:20:53Z" generation: 6 name: default-ipv4-ippool resourceVersion: "1958612285" uid: 012e7091-a9b8-4095-b4c5-ecfa109085d8 spec: allowedUses:

Workload
Tunnel blockSize: 26 cidr: 172.27.0.0/16 ipipMode: Never natOutgoing: true nodeSelector: all() vxlanMode: CrossSubnet

Kube-proxy Mode: iptables

Could you please help on this .

Vishva066 commented 1 month ago

I am also facing the same issue I am using gcp VM instance. I created two busybox pods and tried to ping the other pod from this pod. But it is not working

coutinhop commented 1 month ago

@udhayd @Vishva066 could you provide more information? Do you have any network policy in place (either Calico or plain kubernetes) that could be blocking DNS traffic? Could you provide the output from kubectl describe tigerastatus and also logs from calico-node and tigera operator?

udhayd commented 1 month ago

@coutinhop ,

We haven't enabled network policy , just upgraded CNI from canal 3.23 to calico 3.28 . we have deployed through calico through manifest . I have attached calico-node log ,Please check !

calico-node.log

coutinhop commented 1 month ago

Even though you have VXLAN set to 'crosssubnet' and IPIP set to 'never' in your IP pool, I see these lines in the logs:

2024-07-16 16:00:50.646 [INFO][78] felix/int_dataplane.go 1416: IPIP enabled, starting thread to keep tunnel configuration in sync. [...] 2024-07-16 16:00:50.646 [INFO][78] felix/ipip_mgr.go 84: IPIP thread started.

Can you post your default FelixConfiguration yaml? I suspect it might be overriding the encapsulation to IPIP

udhayd commented 1 month ago

Hi @coutinhop ,

Please find the below mentioned felixconfiguration.

`apiVersion: v1 items:

apiVersion: crd.projectcalico.org/v1 kind: FelixConfiguration metadata: annotations: projectcalico.org/metadata: '{"creationTimestamp":"2024-07-16T05:59:15Z"}' creationTimestamp: "2024-07-16T05:59:15Z" generation: 1 name: default resourceVersion: "1959620587" uid: fb48002a-b5cc-4cf7-bece-8720754555d5 spec: bpfConnectTimeLoadBalancing: TCP bpfHostNetworkedNATWithoutCTLB: Enabled bpfLogLevel: "" floatingIPs: Disabled logSeverityScreen: Info reportingInterval: 0s kind: List metadata: resourceVersion: "" `

udhayd commented 1 month ago

Hi @coutinhop ,

Please find the IPPool configuration,

`apiVersion: v1 items:

apiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: annotations: projectcalico.org/metadata: '{"creationTimestamp":"2024-07-16T05:59:15Z"}' creationTimestamp: "2024-07-16T05:59:15Z" generation: 2 name: default-ipv4-ippool resourceVersion: "1959767775" uid: 9c235caf-9a82-4894-ab55-9033165292ec spec: allowedUses:
- Workload
- Tunnel blockSize: 26 cidr: 172.27.0.0/16 ipipMode: CrossSubnet natOutgoing: true nodeSelector: all() vxlanMode: Never kind: List metadata: resourceVersion: "" `

coutinhop commented 1 month ago

Oh so you do have IPIP enabled! I was scratching my head for a minute heh... Well, then I guess the encapsulation was a red herring and is actually fine...

udhayd commented 1 month ago

@coutinhop ,

Is the above configuration good for calico to work as expected ??

udhayd commented 1 month ago

hi @coutinhop ,

Servicename resolution doesnt happen properly , we have coredns running with 4 replicas two replicas work & remaining two doesnt work. We were not able to conclude issue with specific node , it behaves weirdly .

Due to this , we were not able to access workloads in cluster.

caseydavenport commented 1 month ago

We have upgraded CNI Plugin from canal v3.23 to calico v3.28 in k8s v1.28 cluster

Could you describe the steps / process you took to perform this upgrade? Switching from Canal -> Calico requires following this documentation: https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/migration-from-flannel

Other useful information would be the output of the following commands on the source / destination nodes:

ip route show
ip addr show
iptables-save -c

Finally, switching from VXLAN -> IPIP as part of this migration means that your firewall rules may need to change in order to allow IPIP instead of VXLAN. See here: https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements

udhayd commented 1 month ago

Hi @caseydavenport ,

We have removed canal CNI & deployed Calico v3.28.0 version in Kubernetes v1.28 cluster & dont have firewall enabled on worker nodes.

These are the worker nodes cidr=10.77.4.0/22,10.77.20.0/22,10.77.36.0/22 , should we change the IPIP mode to "CrossSubnet" ??

apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
  kind: IPPool
  metadata:
    annotations:
      projectcalico.org/metadata: '{"creationTimestamp":"2024-07-22T08:04:03Z"}'
    creationTimestamp: "2024-07-22T08:04:03Z"
    generation: 1
    name: default-ipv4-ippool
    resourceVersion: "1965391506"
    uid: 9be50216-c080-4e5c-a8ba-0bfc7b06fec7
  spec:
    allowedUses:
    - Workload
    - Tunnel
    blockSize: 26
    cidr: 172.27.0.0/16
    ipipMode: Always
    natOutgoing: true
    nodeSelector: all()
    vxlanMode: Never
kind: List
metadata:
  resourceVersion: ""

ip route show

    default via 10.77.4.1 dev eth0
    10.77.4.0/22 dev eth0 proto kernel scope link src 10.77.7.185
    10.83.144.0/22 dev eth1 proto kernel scope link src 10.83.147.146
    169.254.0.0/16 dev eth0 scope link metric 1002
    169.254.0.0/16 dev eth1 scope link metric 1003
    172.27.0.0/26 via 10.83.147.147 dev tunl0 proto bird onlink
    172.27.0.192/26 via 10.83.147.147 dev tunl0 proto bird onlink
    172.27.1.64/26 via 10.77.36.62 dev tunl0 proto bird onlink
    172.27.3.64/26 via 10.77.7.182 dev tunl0 proto bird onlink
    172.27.3.192/26 via 10.83.148.45 dev tunl0 proto bird onlink
    172.27.5.128/26 proto bird
        nexthop via 10.77.7.160 dev eth0 weight 1
        nexthop via 10.83.147.123 dev eth1 weight 1
    172.27.6.128/26 proto bird
        nexthop via 10.77.7.161 dev eth0 weight 1
        nexthop via 10.83.147.124 dev eth1 weight 1
    172.27.7.128/26 via 10.83.147.143 dev tunl0 proto bird onlink
    172.27.8.128/26 via 10.83.148.73 dev tunl0 proto bird onlink
    172.27.9.0/26 proto bird
        nexthop via 10.77.7.159 dev eth0 weight 1
        nexthop via 10.83.147.122 dev eth1 weight 1
    unreachable 172.27.10.64/26 proto bird
    172.27.11.128/26 via 10.77.7.161 dev tunl0 proto bird onlink
    172.27.14.128/26 via 10.77.22.19 dev tunl0 proto bird onlink
    172.27.16.128/26 via 10.77.20.204 dev tunl0 proto bird onlink
    172.27.20.128/26 via 10.77.20.209 dev tunl0 proto bird onlink
    172.27.21.64/26 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.29.0/26 via 10.83.154.42 dev tunl0 proto bird onlink
    172.27.30.192/26 via 10.83.153.185 dev tunl0 proto bird onlink
    172.27.31.192/26 via 10.77.22.16 dev tunl0 proto bird onlink
    172.27.32.0/26 via 10.77.20.195 dev tunl0 proto bird onlink
    172.27.32.64/26 via 10.83.153.186 dev tunl0 proto bird onlink
    172.27.32.192/26 via 10.77.36.79 dev tunl0 proto bird onlink
    172.27.33.128/26 via 10.83.145.122 dev tunl0 proto bird onlink
    172.27.34.128/26 via 10.83.154.43 dev tunl0 proto bird onlink
    172.27.35.64/26 via 10.83.145.121 dev tunl0 proto bird onlink
    172.27.36.192/26 via 10.83.153.187 dev tunl0 proto bird onlink
    172.27.37.128/26 via 10.77.20.143 dev tunl0 proto bird onlink
    172.27.38.128/26 via 10.77.20.190 dev tunl0 proto bird onlink
    172.27.39.192/26 via 10.77.20.149 dev tunl0 proto bird onlink
    172.27.40.0/26 via 10.77.20.194 dev tunl0 proto bird onlink
    172.27.41.192/26 via 10.77.20.151 dev tunl0 proto bird onlink
    172.27.42.0/26 via 10.77.20.195 dev tunl0 proto bird onlink
    172.27.43.0/26 via 10.77.20.201 dev tunl0 proto bird onlink
    172.27.44.0/26 via 10.77.20.144 dev tunl0 proto bird onlink
    172.27.45.64/26 via 10.77.20.150 dev tunl0 proto bird onlink
    172.27.46.0/26 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.25 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.26 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.27 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.36 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.38 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.41 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.42 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.43 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.46 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.51 via 10.77.20.202 dev tunl0 proto bird onlink
    172.27.46.192/26 via 10.77.36.64 dev tunl0 proto bird onlink
    172.27.47.0/26 via 10.77.20.198 dev tunl0 proto bird onlink
    172.27.48.128/26 via 10.77.20.196 dev tunl0 proto bird onlink
    172.27.49.0/26 via 10.77.20.217 dev tunl0 proto bird onlink
    172.27.50.64/26 via 10.77.20.145 dev tunl0 proto bird onlink
    172.27.51.0/26 via 10.77.20.210 dev tunl0 proto bird onlink
    172.27.51.64/26 via 10.77.20.210 dev tunl0 proto bird onlink
    172.27.52.64/26 via 10.77.20.209 dev tunl0 proto bird onlink
    172.27.52.192/26 via 10.77.20.194 dev tunl0 proto bird onlink
    172.27.53.64/26 via 10.77.20.207 dev tunl0 proto bird onlink
    172.27.53.128/26 via 10.77.20.207 dev tunl0 proto bird onlink
    172.27.54.64/26 via 10.77.20.206 dev tunl0 proto bird onlink
    172.27.55.0/26 via 10.77.20.212 dev tunl0 proto bird onlink
    172.27.56.64/26 via 10.77.20.204 dev tunl0 proto bird onlink
    172.27.57.128/26 via 10.77.20.203 dev tunl0 proto bird onlink
    172.27.57.192/26 via 10.77.20.143 dev tunl0 proto bird onlink
    172.27.58.0/26 via 10.77.20.143 dev tunl0 proto bird onlink
    172.27.63.192/26 via 10.77.20.212 dev tunl0 proto bird onlink
    172.27.68.0/26 via 10.77.22.148 dev tunl0 proto bird onlink
    172.27.68.23 via 10.77.20.198 dev tunl0 proto bird onlink
    172.27.68.24 via 10.77.20.198 dev tunl0 proto bird onlink
    172.27.68.64/26 via 10.77.20.198 dev tunl0 proto bird onlink
    172.27.68.68 via 10.77.22.148 dev tunl0 proto bird onlink
    172.27.68.69 via 10.77.22.148 dev tunl0 proto bird onlink
    172.27.76.0/26 via 10.77.20.151 dev tunl0 proto bird onlink
    172.27.80.0/26 via 10.77.20.206 dev tunl0 proto bird onlink
    172.27.82.192/26 via 10.77.7.186 dev tunl0 proto bird onlink
    172.27.85.64/26 via 10.77.22.15 dev tunl0 proto bird onlink
    172.27.92.0/26 via 10.77.22.14 dev tunl0 proto bird onlink
    172.27.94.192/26 via 10.77.20.210 dev tunl0 proto bird onlink
    172.27.120.128/26 via 10.77.20.150 dev tunl0 proto bird onlink
    172.27.133.0/26 via 10.77.20.203 dev tunl0 proto bird onlink
    172.27.134.192/26 via 10.77.22.149 dev tunl0 proto bird onlink
    172.27.135.0/26 via 10.77.20.217 dev tunl0 proto bird onlink
    172.27.145.0/26 via 10.77.22.16 dev tunl0 proto bird onlink
    172.27.159.0/26 via 10.77.22.17 dev tunl0 proto bird onlink
    172.27.164.64/26 via 10.77.36.62 dev tunl0 proto bird onlink
    172.27.174.0/26 via 10.77.20.196 dev tunl0 proto bird onlink
    172.27.175.64/26 via 10.77.7.159 dev tunl0 proto bird onlink
    172.27.175.192/26 via 10.77.20.207 dev tunl0 proto bird onlink
    172.27.191.192/26 via 10.77.20.144 dev tunl0 proto bird onlink
    blackhole 172.27.198.192/26 proto bird
    172.27.198.195 dev cali5ce32826680 scope link
    172.27.198.198 dev calia9a07b1e42e scope link
    172.27.198.199 dev cali15ccdd72ef3 scope link
    172.27.198.201 dev cali5067fa7b421 scope link
    172.27.198.209 dev calia5fdf11a092 scope link
    172.27.198.218 dev cali76c44cc8dbc scope link
    172.27.198.219 dev caliac9151b0f10 scope link
    172.27.198.221 dev cali0545b436a57 scope link
    172.27.198.225 via 10.77.20.145 dev tunl0 proto bird onlink
    172.27.198.226 via 10.77.20.145 dev tunl0 proto bird onlink
    172.27.198.229 dev cali879ff228c90 scope link
    172.27.198.230 dev cali58c9cd1c84d scope link
    172.27.198.231 dev cali4d63d0c25a4 scope link
    172.27.198.232 dev cali9bb0bda72cb scope link
    172.27.198.234 dev cali098a73f6489 scope link
    172.27.198.236 dev cali3fe5cc8fa58 scope link
    172.27.198.239 dev cali98335a9cf36 scope link
    172.27.198.240 dev califc0f1f81373 scope link
    172.27.198.241 dev calibf0e1f2be3f scope link
    172.27.198.242 dev cali5aa2eb5ad2b scope link
    172.27.198.243 dev cali07ab5919610 scope link
    172.27.198.244 dev cali4b825fe75ee scope link
    172.27.198.249 dev cali2e813d30fe0 scope link
    172.27.198.250 dev calid73d1c97196 scope link
    172.27.198.251 dev caliec7e287e053 scope link
    172.27.198.252 dev cali9bd3406dd1f scope link
    172.27.199.0/26 via 10.77.20.145 dev tunl0 proto bird onlink
    172.27.199.39 dev calia1d5bf83389 scope link
    172.27.199.40 dev cali7208e688104 scope link
    172.27.210.0/26 via 10.77.20.190 dev tunl0 proto bird onlink
    172.27.212.0/26 via 10.77.22.18 dev tunl0 proto bird onlink
    172.27.228.0/26 via 10.77.20.149 dev tunl0 proto bird onlink
    172.27.236.192/26 via 10.77.7.160 dev tunl0 proto bird onlink
    172.27.240.0/26 via 10.77.20.201 dev tunl0 proto bird onlink

ip addr show

    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
        inet 127.0.0.1/8 scope host lo
           valid_lft forever preferred_lft forever
    2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:94:39:51 brd ff:ff:ff:ff:ff:ff
        inet 10.77.7.185/22 brd 10.77.7.255 scope global eth0
           valid_lft forever preferred_lft forever
    3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
        link/ether 00:50:56:94:27:eb brd ff:ff:ff:ff:ff:ff
        inet 10.83.147.146/22 brd 10.83.147.255 scope global eth1
           valid_lft forever preferred_lft forever
    4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
        link/ether 00:50:56:94:68:b3 brd ff:ff:ff:ff:ff:ff
    5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
        link/ipip 0.0.0.0 brd 0.0.0.0
        inet 172.27.198.192/32 scope global tunl0
           valid_lft forever preferred_lft forever
    49: calia1d5bf83389@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
    50: cali7208e688104@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 14
    76: cali5ce32826680@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
    79: calia9a07b1e42e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
    80: cali15ccdd72ef3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 8
    82: cali5067fa7b421@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
    90: calia5fdf11a092@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4
    99: cali76c44cc8dbc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 18
    100: caliac9151b0f10@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 13
    102: cali0545b436a57@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 16
    110: cali879ff228c90@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
    111: cali58c9cd1c84d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 12
    112: cali4d63d0c25a4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 15
    113: cali9bb0bda72cb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 19
    115: cali098a73f6489@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 21
    120: cali98335a9cf36@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 26
    121: califc0f1f81373@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 27
    122: calibf0e1f2be3f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 28
    124: cali07ab5919610@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 29
    125: cali4b825fe75ee@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 9
    130: cali2e813d30fe0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 20
    131: calid73d1c97196@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 6
    132: caliec7e287e053@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 22
    133: cali9bd3406dd1f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 11
    160: cali3fe5cc8fa58@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 10
    163: cali5aa2eb5ad2b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
        link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0

caseydavenport commented 1 month ago

    172.27.5.128/26 proto bird
        nexthop via 10.77.7.160 dev eth0 weight 1
        nexthop via 10.83.147.123 dev eth1 weight 1

These ECMP blocks in your routing table are fishy - typically Calico won't create routes like this for pod IP blocks in a healthy system, and it suggests that two nodes think they own the same /26 which is likely to cause problems.

We have removed canal CNI & deployed Calico v3.28.0 version in Kubernetes v1.28 cluster & dont have firewall enabled on worker nodes.

Simply deleting Canal and Installing Calico is likely to cause problems without following the migration procedure I linked in my previous comment. Hard to say what state you may be in. The simplest thing to do would be to effectively restart the cluster - removing any vestiges of Canal from the node (e.g., in /etc/cni/net.d) and then restart each node in turn to ensure a fresh slate.

It's possible that leftover state from the Canal installation is resulting in the duplicated routes above.

dont have firewall enabled on worker nodes.

Where is this cluster running? e.g., typically in GCP / AWS you would need to configure cloud firewall rules / security groups.

following commands on the source / destination nodes:

Which node is the above output coming from?

udhayd commented 1 month ago

@caseydavenport ,

Thanks for your help . I have perform network cleanup on all nodes , Issue has been fixed . It works now.

projectcalico / calico