search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.73k stars 1.28k forks source link

large pod packets is not encapsulated in ip-in-ip packets and are silently dropped #8953

Open ImadNaily opened 2 weeks ago

ImadNaily commented 2 weeks ago

I have several pods running on a host, lets say host1, listening on https ports. I can connect to all those pods from a host, lets say host3, using openssl s_client -connect podIP:port with no issues.

However, from another host, lets say host2, i cannot.

Looking at the tcpdump output when connecting from host2, i can see that the return traffic from the pods is not encapsulated in IP-in-IP traffic per below:

11:58:17.439622 IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1:2395, ack 290, win 226, options [nop,nop,TS val 1169287868 ecr 1258496083], length 2394
11:58:17.439630 IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1:2395, ack 290, win 226, options [nop,nop,TS val 1169287868 ecr 1258496083], length 2394
11:58:17.449788 IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1389:2395, ack 290, win 226, options [nop,nop,TS val 1169287879 ecr 1258496083], length 1006
11:58:17.449800 IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1389:2395, ack 290, win 226, options [nop,nop,TS val 1169287879 ecr 1258496083], length 1006
11:58:17.449806 IP 10.100.30.42 > 10.220.10.32: IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1389:2395, ack 290, win 226, options [nop,nop,TS val 1169287879 ecr 1258496083], length 1006 (ipip-proto-4)
11:58:17.449809 IP 10.100.30.42 > 10.220.10.32: IP 10.96.180.199.5000 > 10.96.66.192.49360: Flags [P.], seq 1389:2395, ack 290, win 226, options [nop,nop,TS val 1169287879 ecr 1258496083], length 1006 (ipip-proto-4)
1

the above clearly shows that the packet with 2394 length is not encapsulated in IP-in-IP and is silently dropped, yet the following packet which is 1006 in length is, as seen by the following tcpdump IP-in-IP packets.

the tcpdump from host3 shows that the large packets are encapsulated with no problem.

Expected Behavior

the large packets from the pod should be encapsulated in IP-in-IP regardless.

Current Behavior

since this is ssl connection, the handshake is not completed when connecting from host2.

Possible Solution

no idea

Steps to Reproduce (for bugs)

Context

trying to connect to a pod from host2 using openssl

Your Environment

Client Version: v3.24.3 Git commit: d833a9e38 Cluster Version: v3.24.3 Cluster Type: k8s,bgp,kubeadm,kdd

kubadm version Client Version: v1.25.0 Server Version: v1.25.3

caseydavenport commented 2 weeks ago

Could you share the routing table on each of the source nodes (host2 / host3)? That is what I would expect would determine whether a packet goes over the tunnel or not.

fasaxc commented 1 week ago

what is the MTU set on the IPIP tunnel on each of the hosts? What is the MTU on the eth0?

ip link show

should show the MTU. Calico will attempt to auto-detect the MTU of your "eth0" device at startup and applyt eh correct value to the tunnel.

ImadNaily commented 1 week ago

All cali and tunnel interfaces have MTU of 1440 on all hosts the eth0 MTU is 1550 note that the ip of the pod that i cannot connect to is: 10.96.180.197 thank you for the help

the ip route for the host that cannot connect is:

 ip route
default via 10.220.10.250 dev ifcfg-team0 proto static metric 350
10.96.5.128/26 via 10.100.99.22 dev tunl0 proto bird onlink
10.96.7.128/26 via 10.220.10.31 dev tunl0 proto bird onlink
10.96.7.192/26 via 10.220.10.31 dev tunl0 proto bird onlink
10.96.52.192/26 via 10.230.10.20 dev tunl0 proto bird onlink
blackhole 10.96.66.192/26 proto bird
10.96.66.195 dev cali08d091557e9 scope link
10.96.66.196 dev calic62e4959c17 scope link
10.96.66.197 dev cali90ed1524e04 scope link
10.96.66.198 dev cali85478fd1c02 scope link
10.96.66.199 dev calib2f55d4637b scope link
10.96.66.200 dev cali65e5e7353bc scope link
10.96.66.204 dev cali078077c6d11 scope link
10.96.66.206 dev cali9b13c26dc4d scope link
10.96.66.208 dev cali7f5a13d4712 scope link
10.96.66.209 dev cali6c627deb8ea scope link
10.96.66.210 dev calif44990189cd scope link
10.96.66.211 dev calic2f4295b964 scope link
10.96.66.214 dev cali2bcf345eacd scope link
10.96.66.215 dev cali403f8bbccfc scope link
10.96.66.216 dev cali7b9483bea1e scope link
10.96.66.217 dev cali8bb5c159573 scope link
10.96.66.218 dev cali2b6089a3a1f scope link
10.96.66.219 dev calidb08c1ac1c3 scope link
10.96.66.220 dev cali6407e5b5fab scope link
10.96.66.221 dev calif10fb6dcb90 scope link
10.96.66.222 dev calid5189036b34 scope link
10.96.66.223 dev cali08f6ba3e580 scope link
10.96.66.224 dev caliefc6516d8a8 scope link
10.96.66.226 dev cali9830798afc2 scope link
10.96.66.228 dev cali8c89279eae7 scope link
10.96.66.230 dev cali056ba866619 scope link
10.96.66.232 dev cali6957d1ce682 scope link
10.96.66.234 dev cali7081257756a scope link
10.96.66.235 dev calic999161b70c scope link
10.96.66.237 dev cali7a234aaf9d8 scope link
10.96.66.239 dev cali533af98cd43 scope link
10.96.66.240 dev cali8c950601efe scope link
10.96.66.243 dev cali6a34e0ac97b scope link
10.96.66.244 dev cali68bc9f3baee scope link
10.96.66.247 dev calidf9d4622547 scope link
10.96.66.248 dev caliabc394bbfec scope link
10.96.66.249 dev calicd57cf86f78 scope link
10.96.66.251 dev cali94730a07e30 scope link
10.96.66.252 dev cali5390be63cde scope link
10.96.66.253 dev cali469984e0ca7 scope link
10.96.66.254 dev cali09633d8311f scope link
10.96.66.255 dev cali20382e5ffdf scope link
10.96.98.128/26 via 192.168.33.110 dev tunl0 proto bird onlink
10.96.105.64/26 via 10.100.30.49 dev tunl0 proto bird onlink
10.96.130.64/26 via 10.210.20.30 dev tunl0 proto bird onlink
10.96.151.192/26 via 10.230.19.24 dev tunl0 proto bird onlink
10.96.169.128/26 via 10.230.10.21 dev tunl0 proto bird onlink
10.96.180.192/26 via 10.100.30.42 dev tunl0 proto bird onlink
10.96.181.0/26 via 10.100.30.42 dev tunl0 proto bird onlink
10.96.198.128/26 via 10.100.99.21 dev tunl0 proto bird onlink
10.96.200.0/26 via 10.100.30.41 dev tunl0 proto bird onlink
10.96.200.64/26 via 10.100.30.41 dev tunl0 proto bird onlink
10.96.205.192/26 via 10.100.4.10 dev tunl0 proto bird onlink
10.220.10.0/24 dev ifcfg-team0 proto kernel scope link src 10.220.10.32 metric 350
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

the ip route for the host that can connect:

default via 10.220.10.250 dev team0 proto static metric 350
10.96.5.128/26 via 10.100.99.22 dev tunl0 proto bird onlink
10.96.7.128 dev cali342b9176357 scope link
blackhole 10.96.7.128/26 proto bird
10.96.7.129 dev calib318192e929 scope link
10.96.7.130 dev cali3737ac1b4f7 scope link
10.96.7.131 dev calid298c64c223 scope link
10.96.7.132 dev cali69b9dd69e00 scope link
10.96.7.133 dev cali61948faaa20 scope link
10.96.7.134 dev califb8684771fd scope link
10.96.7.135 dev calid37d6142daa scope link
10.96.7.136 dev cali4d77c6aee54 scope link
10.96.7.137 dev calif69307e6cad scope link
10.96.7.138 dev califc43e95b53d scope link
10.96.7.139 dev cali849891b5f87 scope link
10.96.7.140 dev calid09b85f72ef scope link
10.96.7.141 dev cali9983b447eab scope link
10.96.7.142 dev cali255b2a36f12 scope link
10.96.7.143 dev cali807eccc34a7 scope link
10.96.7.144 dev calibb87ea169cf scope link
10.96.7.145 dev cali340dc3168ca scope link
10.96.7.146 dev cali0a268e607f4 scope link
10.96.7.147 dev cali94822b45c8e scope link
10.96.7.148 dev cali1941c41cfa2 scope link
10.96.7.149 dev cali64d81a21dd6 scope link
10.96.7.150 dev calic175e16328e scope link
10.96.7.151 dev cali017fb993fab scope link
10.96.7.152 dev calibbdf14478b9 scope link
10.96.7.154 dev cali382377d9fdb scope link
10.96.7.155 dev calic9592a4e276 scope link
10.96.7.156 dev caliebffa0d992e scope link
10.96.7.157 dev cali9b4b89f77ce scope link
10.96.7.158 dev calied2f17f0cea scope link
10.96.7.159 dev calieb3a9f849e9 scope link
10.96.7.160 dev cali4ead294458c scope link
10.96.7.161 dev cali35dba197ec7 scope link
10.96.7.162 dev calia09a1c6b61e scope link
10.96.7.163 dev cali49dd2fe09fa scope link
10.96.7.164 dev caliebd53bf40b7 scope link
10.96.7.165 dev cali57a82e0f507 scope link
10.96.7.166 dev cali932d9151431 scope link
10.96.7.167 dev cali56b37315788 scope link
10.96.7.168 dev califfb9ba2dfc3 scope link
10.96.7.169 dev cali2795802ea57 scope link
10.96.7.170 dev calidb959c9b314 scope link
10.96.7.171 dev calia2d4ff702b4 scope link
10.96.7.172 dev calib060aa643b8 scope link
10.96.7.174 dev calid3d954817f5 scope link
10.96.7.175 dev calif3621e8659f scope link
10.96.7.176 dev cali7da87be0ae9 scope link
10.96.7.177 dev cali5d07ed929e1 scope link
10.96.7.178 dev cali3a9c6a79f4b scope link
10.96.7.179 dev calif5092251a88 scope link
10.96.7.180 dev cali559d017d942 scope link
10.96.7.181 dev cali40d198873f7 scope link
10.96.7.182 dev cali9662095dc57 scope link
10.96.7.183 dev cali9744aec07a2 scope link
10.96.7.184 dev cali316ca45a313 scope link
10.96.7.185 dev cali7b2dd5d6f68 scope link
10.96.7.186 dev cali8bb9bcf7ea6 scope link
10.96.7.187 dev cali3943b488c3a scope link
10.96.7.188 dev cali4877d276b1f scope link
10.96.7.189 dev cali760ec3098c4 scope link
10.96.7.190 dev calicc32cdfa2f0 scope link
10.96.7.191 dev cali3a56617df61 scope link
10.96.7.192 dev cali70b5f632a15 scope link
blackhole 10.96.7.192/26 proto bird
10.96.7.196 dev cali896211fcd5a scope link
10.96.7.197 dev calib7d4d83f79e scope link
10.96.7.198 dev cali2b902584c39 scope link
10.96.7.199 dev cali279834351c1 scope link
10.96.7.200 dev cali0f6ffd8942b scope link
10.96.7.201 dev calid1a10002517 scope link
10.96.7.202 dev cali90be418b548 scope link
10.96.7.203 dev cali5e08a030757 scope link
10.96.7.204 dev cali0fd5610d209 scope link
10.96.7.206 dev cali5e7a6d480ae scope link
10.96.7.223 dev calif4029ccb91b scope link
10.96.7.229 dev cali53ab4257902 scope link
10.96.7.230 dev calie0800a644a9 scope link
10.96.7.236 dev cali229ea498b52 scope link
10.96.7.237 dev calif93fcf3aceb scope link
10.96.7.238 dev cali37b41d94a32 scope link
10.96.52.192/26 via 10.230.10.20 dev tunl0 proto bird onlink
10.96.66.192/26 via 10.220.10.32 dev tunl0 proto bird onlink
10.96.98.128/26 via 192.168.33.110 dev tunl0 proto bird onlink
10.96.105.64/26 via 10.100.30.49 dev tunl0 proto bird onlink
10.96.130.64/26 via 10.210.20.30 dev tunl0 proto bird onlink
10.96.151.192/26 via 10.230.19.24 dev tunl0 proto bird onlink
10.96.169.128/26 via 10.230.10.21 dev tunl0 proto bird onlink
10.96.180.192/26 via 10.100.30.42 dev tunl0 proto bird onlink
10.96.181.0/26 via 10.100.30.42 dev tunl0 proto bird onlink
10.96.198.128/26 via 10.100.99.21 dev tunl0 proto bird onlink
10.96.200.0/26 via 10.100.30.41 dev tunl0 proto bird onlink
10.96.200.64/26 via 10.100.30.41 dev tunl0 proto bird onlink
10.96.205.192/26 via 10.100.4.10 dev tunl0 proto bird onlink
10.220.10.0/24 dev team0 proto kernel scope link src 10.220.10.31 metric 350
169.254.0.0/16 dev eno1 scope link metric 1002
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1