Closed udhayd closed 1 month ago
I am also facing the same issue I am using gcp VM instance. I created two busybox pods and tried to ping the other pod from this pod. But it is not working
@udhayd @Vishva066 could you provide more information? Do you have any network policy in place (either Calico or plain kubernetes) that could be blocking DNS traffic? Could you provide the output from kubectl describe tigerastatus
and also logs from calico-node and tigera operator?
@coutinhop ,
We haven't enabled network policy , just upgraded CNI from canal 3.23 to calico 3.28 . we have deployed through calico through manifest . I have attached calico-node log ,Please check !
Even though you have VXLAN set to 'crosssubnet' and IPIP set to 'never' in your IP pool, I see these lines in the logs:
2024-07-16 16:00:50.646 [INFO][78] felix/int_dataplane.go 1416: IPIP enabled, starting thread to keep tunnel configuration in sync. [...] 2024-07-16 16:00:50.646 [INFO][78] felix/ipip_mgr.go 84: IPIP thread started.
Can you post your default FelixConfiguration yaml? I suspect it might be overriding the encapsulation to IPIP
Hi @coutinhop ,
Please find the below mentioned felixconfiguration.
`apiVersion: v1 items:
Hi @coutinhop ,
Please find the IPPool configuration,
`apiVersion: v1 items:
Oh so you do have IPIP enabled! I was scratching my head for a minute heh... Well, then I guess the encapsulation was a red herring and is actually fine...
@coutinhop ,
Is the above configuration good for calico to work as expected ??
hi @coutinhop ,
Servicename resolution doesnt happen properly , we have coredns running with 4 replicas two replicas work & remaining two doesnt work. We were not able to conclude issue with specific node , it behaves weirdly .
Due to this , we were not able to access workloads in cluster.
We have upgraded CNI Plugin from canal v3.23 to calico v3.28 in k8s v1.28 cluster
Could you describe the steps / process you took to perform this upgrade? Switching from Canal -> Calico requires following this documentation: https://docs.tigera.io/calico/latest/getting-started/kubernetes/flannel/migration-from-flannel
Other useful information would be the output of the following commands on the source / destination nodes:
ip route show
ip addr show
iptables-save -c
Finally, switching from VXLAN -> IPIP as part of this migration means that your firewall rules may need to change in order to allow IPIP instead of VXLAN. See here: https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements
Hi @caseydavenport ,
We have removed canal CNI & deployed Calico v3.28.0 version in Kubernetes v1.28 cluster & dont have firewall enabled on worker nodes.
These are the worker nodes cidr=10.77.4.0/22,10.77.20.0/22,10.77.36.0/22 , should we change the IPIP mode to "CrossSubnet" ??
apiVersion: v1
items:
- apiVersion: crd.projectcalico.org/v1
kind: IPPool
metadata:
annotations:
projectcalico.org/metadata: '{"creationTimestamp":"2024-07-22T08:04:03Z"}'
creationTimestamp: "2024-07-22T08:04:03Z"
generation: 1
name: default-ipv4-ippool
resourceVersion: "1965391506"
uid: 9be50216-c080-4e5c-a8ba-0bfc7b06fec7
spec:
allowedUses:
- Workload
- Tunnel
blockSize: 26
cidr: 172.27.0.0/16
ipipMode: Always
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
kind: List
metadata:
resourceVersion: ""
ip route show
default via 10.77.4.1 dev eth0
10.77.4.0/22 dev eth0 proto kernel scope link src 10.77.7.185
10.83.144.0/22 dev eth1 proto kernel scope link src 10.83.147.146
169.254.0.0/16 dev eth0 scope link metric 1002
169.254.0.0/16 dev eth1 scope link metric 1003
172.27.0.0/26 via 10.83.147.147 dev tunl0 proto bird onlink
172.27.0.192/26 via 10.83.147.147 dev tunl0 proto bird onlink
172.27.1.64/26 via 10.77.36.62 dev tunl0 proto bird onlink
172.27.3.64/26 via 10.77.7.182 dev tunl0 proto bird onlink
172.27.3.192/26 via 10.83.148.45 dev tunl0 proto bird onlink
172.27.5.128/26 proto bird
nexthop via 10.77.7.160 dev eth0 weight 1
nexthop via 10.83.147.123 dev eth1 weight 1
172.27.6.128/26 proto bird
nexthop via 10.77.7.161 dev eth0 weight 1
nexthop via 10.83.147.124 dev eth1 weight 1
172.27.7.128/26 via 10.83.147.143 dev tunl0 proto bird onlink
172.27.8.128/26 via 10.83.148.73 dev tunl0 proto bird onlink
172.27.9.0/26 proto bird
nexthop via 10.77.7.159 dev eth0 weight 1
nexthop via 10.83.147.122 dev eth1 weight 1
unreachable 172.27.10.64/26 proto bird
172.27.11.128/26 via 10.77.7.161 dev tunl0 proto bird onlink
172.27.14.128/26 via 10.77.22.19 dev tunl0 proto bird onlink
172.27.16.128/26 via 10.77.20.204 dev tunl0 proto bird onlink
172.27.20.128/26 via 10.77.20.209 dev tunl0 proto bird onlink
172.27.21.64/26 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.29.0/26 via 10.83.154.42 dev tunl0 proto bird onlink
172.27.30.192/26 via 10.83.153.185 dev tunl0 proto bird onlink
172.27.31.192/26 via 10.77.22.16 dev tunl0 proto bird onlink
172.27.32.0/26 via 10.77.20.195 dev tunl0 proto bird onlink
172.27.32.64/26 via 10.83.153.186 dev tunl0 proto bird onlink
172.27.32.192/26 via 10.77.36.79 dev tunl0 proto bird onlink
172.27.33.128/26 via 10.83.145.122 dev tunl0 proto bird onlink
172.27.34.128/26 via 10.83.154.43 dev tunl0 proto bird onlink
172.27.35.64/26 via 10.83.145.121 dev tunl0 proto bird onlink
172.27.36.192/26 via 10.83.153.187 dev tunl0 proto bird onlink
172.27.37.128/26 via 10.77.20.143 dev tunl0 proto bird onlink
172.27.38.128/26 via 10.77.20.190 dev tunl0 proto bird onlink
172.27.39.192/26 via 10.77.20.149 dev tunl0 proto bird onlink
172.27.40.0/26 via 10.77.20.194 dev tunl0 proto bird onlink
172.27.41.192/26 via 10.77.20.151 dev tunl0 proto bird onlink
172.27.42.0/26 via 10.77.20.195 dev tunl0 proto bird onlink
172.27.43.0/26 via 10.77.20.201 dev tunl0 proto bird onlink
172.27.44.0/26 via 10.77.20.144 dev tunl0 proto bird onlink
172.27.45.64/26 via 10.77.20.150 dev tunl0 proto bird onlink
172.27.46.0/26 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.25 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.26 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.27 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.36 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.38 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.41 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.42 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.43 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.46 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.51 via 10.77.20.202 dev tunl0 proto bird onlink
172.27.46.192/26 via 10.77.36.64 dev tunl0 proto bird onlink
172.27.47.0/26 via 10.77.20.198 dev tunl0 proto bird onlink
172.27.48.128/26 via 10.77.20.196 dev tunl0 proto bird onlink
172.27.49.0/26 via 10.77.20.217 dev tunl0 proto bird onlink
172.27.50.64/26 via 10.77.20.145 dev tunl0 proto bird onlink
172.27.51.0/26 via 10.77.20.210 dev tunl0 proto bird onlink
172.27.51.64/26 via 10.77.20.210 dev tunl0 proto bird onlink
172.27.52.64/26 via 10.77.20.209 dev tunl0 proto bird onlink
172.27.52.192/26 via 10.77.20.194 dev tunl0 proto bird onlink
172.27.53.64/26 via 10.77.20.207 dev tunl0 proto bird onlink
172.27.53.128/26 via 10.77.20.207 dev tunl0 proto bird onlink
172.27.54.64/26 via 10.77.20.206 dev tunl0 proto bird onlink
172.27.55.0/26 via 10.77.20.212 dev tunl0 proto bird onlink
172.27.56.64/26 via 10.77.20.204 dev tunl0 proto bird onlink
172.27.57.128/26 via 10.77.20.203 dev tunl0 proto bird onlink
172.27.57.192/26 via 10.77.20.143 dev tunl0 proto bird onlink
172.27.58.0/26 via 10.77.20.143 dev tunl0 proto bird onlink
172.27.63.192/26 via 10.77.20.212 dev tunl0 proto bird onlink
172.27.68.0/26 via 10.77.22.148 dev tunl0 proto bird onlink
172.27.68.23 via 10.77.20.198 dev tunl0 proto bird onlink
172.27.68.24 via 10.77.20.198 dev tunl0 proto bird onlink
172.27.68.64/26 via 10.77.20.198 dev tunl0 proto bird onlink
172.27.68.68 via 10.77.22.148 dev tunl0 proto bird onlink
172.27.68.69 via 10.77.22.148 dev tunl0 proto bird onlink
172.27.76.0/26 via 10.77.20.151 dev tunl0 proto bird onlink
172.27.80.0/26 via 10.77.20.206 dev tunl0 proto bird onlink
172.27.82.192/26 via 10.77.7.186 dev tunl0 proto bird onlink
172.27.85.64/26 via 10.77.22.15 dev tunl0 proto bird onlink
172.27.92.0/26 via 10.77.22.14 dev tunl0 proto bird onlink
172.27.94.192/26 via 10.77.20.210 dev tunl0 proto bird onlink
172.27.120.128/26 via 10.77.20.150 dev tunl0 proto bird onlink
172.27.133.0/26 via 10.77.20.203 dev tunl0 proto bird onlink
172.27.134.192/26 via 10.77.22.149 dev tunl0 proto bird onlink
172.27.135.0/26 via 10.77.20.217 dev tunl0 proto bird onlink
172.27.145.0/26 via 10.77.22.16 dev tunl0 proto bird onlink
172.27.159.0/26 via 10.77.22.17 dev tunl0 proto bird onlink
172.27.164.64/26 via 10.77.36.62 dev tunl0 proto bird onlink
172.27.174.0/26 via 10.77.20.196 dev tunl0 proto bird onlink
172.27.175.64/26 via 10.77.7.159 dev tunl0 proto bird onlink
172.27.175.192/26 via 10.77.20.207 dev tunl0 proto bird onlink
172.27.191.192/26 via 10.77.20.144 dev tunl0 proto bird onlink
blackhole 172.27.198.192/26 proto bird
172.27.198.195 dev cali5ce32826680 scope link
172.27.198.198 dev calia9a07b1e42e scope link
172.27.198.199 dev cali15ccdd72ef3 scope link
172.27.198.201 dev cali5067fa7b421 scope link
172.27.198.209 dev calia5fdf11a092 scope link
172.27.198.218 dev cali76c44cc8dbc scope link
172.27.198.219 dev caliac9151b0f10 scope link
172.27.198.221 dev cali0545b436a57 scope link
172.27.198.225 via 10.77.20.145 dev tunl0 proto bird onlink
172.27.198.226 via 10.77.20.145 dev tunl0 proto bird onlink
172.27.198.229 dev cali879ff228c90 scope link
172.27.198.230 dev cali58c9cd1c84d scope link
172.27.198.231 dev cali4d63d0c25a4 scope link
172.27.198.232 dev cali9bb0bda72cb scope link
172.27.198.234 dev cali098a73f6489 scope link
172.27.198.236 dev cali3fe5cc8fa58 scope link
172.27.198.239 dev cali98335a9cf36 scope link
172.27.198.240 dev califc0f1f81373 scope link
172.27.198.241 dev calibf0e1f2be3f scope link
172.27.198.242 dev cali5aa2eb5ad2b scope link
172.27.198.243 dev cali07ab5919610 scope link
172.27.198.244 dev cali4b825fe75ee scope link
172.27.198.249 dev cali2e813d30fe0 scope link
172.27.198.250 dev calid73d1c97196 scope link
172.27.198.251 dev caliec7e287e053 scope link
172.27.198.252 dev cali9bd3406dd1f scope link
172.27.199.0/26 via 10.77.20.145 dev tunl0 proto bird onlink
172.27.199.39 dev calia1d5bf83389 scope link
172.27.199.40 dev cali7208e688104 scope link
172.27.210.0/26 via 10.77.20.190 dev tunl0 proto bird onlink
172.27.212.0/26 via 10.77.22.18 dev tunl0 proto bird onlink
172.27.228.0/26 via 10.77.20.149 dev tunl0 proto bird onlink
172.27.236.192/26 via 10.77.7.160 dev tunl0 proto bird onlink
172.27.240.0/26 via 10.77.20.201 dev tunl0 proto bird onlink
ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:94:39:51 brd ff:ff:ff:ff:ff:ff
inet 10.77.7.185/22 brd 10.77.7.255 scope global eth0
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 00:50:56:94:27:eb brd ff:ff:ff:ff:ff:ff
inet 10.83.147.146/22 brd 10.83.147.255 scope global eth1
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 00:50:56:94:68:b3 brd ff:ff:ff:ff:ff:ff
5: tunl0@NONE: <NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue state UNKNOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
inet 172.27.198.192/32 scope global tunl0
valid_lft forever preferred_lft forever
49: calia1d5bf83389@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 7
50: cali7208e688104@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 14
76: cali5ce32826680@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 2
79: calia9a07b1e42e@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 5
80: cali15ccdd72ef3@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 8
82: cali5067fa7b421@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 3
90: calia5fdf11a092@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 4
99: cali76c44cc8dbc@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 18
100: caliac9151b0f10@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 13
102: cali0545b436a57@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 16
110: cali879ff228c90@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 1
111: cali58c9cd1c84d@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 12
112: cali4d63d0c25a4@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 15
113: cali9bb0bda72cb@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 19
115: cali098a73f6489@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 21
120: cali98335a9cf36@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 26
121: califc0f1f81373@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 27
122: calibf0e1f2be3f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 28
124: cali07ab5919610@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 29
125: cali4b825fe75ee@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 9
130: cali2e813d30fe0@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 20
131: calid73d1c97196@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 6
132: caliec7e287e053@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 22
133: cali9bd3406dd1f@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 11
160: cali3fe5cc8fa58@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 10
163: cali5aa2eb5ad2b@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default qlen 1000
link/ether ee:ee:ee:ee:ee:ee brd ff:ff:ff:ff:ff:ff link-netnsid 0
172.27.5.128/26 proto bird
nexthop via 10.77.7.160 dev eth0 weight 1
nexthop via 10.83.147.123 dev eth1 weight 1
These ECMP blocks in your routing table are fishy - typically Calico won't create routes like this for pod IP blocks in a healthy system, and it suggests that two nodes think they own the same /26
which is likely to cause problems.
We have removed canal CNI & deployed Calico v3.28.0 version in Kubernetes v1.28 cluster & dont have firewall enabled on worker nodes.
Simply deleting Canal and Installing Calico is likely to cause problems without following the migration procedure I linked in my previous comment. Hard to say what state you may be in. The simplest thing to do would be to effectively restart the cluster - removing any vestiges of Canal from the node (e.g., in /etc/cni/net.d
) and then restart each node in turn to ensure a fresh slate.
It's possible that leftover state from the Canal installation is resulting in the duplicated routes above.
dont have firewall enabled on worker nodes.
Where is this cluster running? e.g., typically in GCP / AWS you would need to configure cloud firewall rules / security groups.
following commands on the source / destination nodes:
Which node is the above output coming from?
@caseydavenport ,
Thanks for your help . I have perform network cleanup on all nodes , Issue has been fixed . It works now.
Expected Behavior
To have pod to pod communication NA ## Current Behavior We have upgraded CNI Plugin from canal v3.23 to calico v3.28 in k8s v1.28 cluster, after deployment of calico v3.28 plugin pod to pod communication didnt work. Pods were not able to reach coredns getting i/o timeout. Calico kube-controllers & calico-nodes are up & running fine. However , we are facing this issue. **Pod Error to coredns:** communications error to 172.27.3.207#53: timed out NA ## Possible SolutionSteps to Reproduce (for bugs)
1. 2. 3. 4.
Context
Due to this issue , we were not able to access the workloads.Your Environment
IP Pool:
apiVersion: crd.projectcalico.org/v1 kind: IPPool metadata: annotations: projectcalico.org/metadata: '{"uid":"68c046a0-6f6a-4fd4-9207-b47a98b40e4a","creationTimestamp":"2022-11-08T09:20:53Z"}' creationTimestamp: "2022-11-08T09:20:53Z" generation: 6 name: default-ipv4-ippool resourceVersion: "1958612285" uid: 012e7091-a9b8-4095-b4c5-ecfa109085d8 spec: allowedUses:
Kube-proxy Mode: iptables
Could you please help on this .