search

flannel-io / flannel

flannel is a network fabric for containers, designed for Kubernetes
Apache License 2.0
8.72k stars 2.87k forks source link

UDP: bad checksum on VXLAN interface #1279

Closed dmitry-irtegov closed 1 year ago

dmitry-irtegov commented 4 years ago

On k8s 1.17 cluster with RHeL 7 nodes, service IPs for pods on other nodes are not accessible. Pod IP seem to work fine. Most noticeably, CoreDNS does not work. Target node dmesg is filled with messages like:

[ 1423.035722] UDP: bad checksum. From 172.16.9.99:27503 to 172.16.80.252:8472 ulen 75
[ 1426.036873] UDP: bad checksum. From 172.16.9.99:4894 to 172.16.80.252:8472 ulen 75
[ 1427.537392] UDP: bad checksum. From 172.16.9.99:32693 to 172.16.80.252:8472 ulen 75
[ 1429.037910] UDP: bad checksum. From 172.16.9.99:46133 to 172.16.80.252:8472 ulen 75

Turning IP checksum offloading on the flannel.1 interface fixes the issue:

[root@ip-172-16-102-241 ~]# nslookup www.google.com 100.64.0.10
;; connection timed out; no servers could be reached
[root@ip-172-16-102-241 ~]# ethtool -K flannel.1 tx-checksum-ip-generic off
Actual changes:
tx-checksumming: off
    tx-checksum-ip-generic: off
tcp-segmentation-offload: off
    tx-tcp-segmentation: off [requested on]
    tx-tcp-ecn-segmentation: off [requested on]
    tx-tcp6-segmentation: off [requested on]
    tx-tcp-mangleid-segmentation: off [requested on]
udp-fragmentation-offload: off [requested on]
[root@ip-172-16-102-241 ~]# nslookup www.google.com 100.64.0.10
Server:     100.64.0.10
Address:    100.64.0.10#53
Non-authoritative answer:
Name:   www.google.com
Address: 172.217.13.228
Name:   www.google.com
Address: 2607:f8b0:4004:80a::2004
[root@ip-172-16-102-241 ~]#

Other people also hit this: https://t.du9l.com/2020/03/kubernetes-flannel-udp-packets-dropped-for-wrong-checksum-workaround/

This happens both with cni-canal and pure cni-flannel, so we decided to report the issue here.

Expected Behavior

I do not have to adjust interface settings to get flannel to work.

Your Environment

CMajeri commented 4 years ago

We hit this too. It was an absolute pain to figure out.

It seems to only affect service IPs (I'm guessing because of masquerade??), and specifically UDP (nslookup doesn't work, nslookup in tcp mode does)

If anyone knows where this bug comes from (besides checksum offloading) I'd be very interested. I'll keep looking for a bit, but I'm not great with networking.

Our environment:

Weird thing is we're running almost the same versions of things (etcd is different, but I really doubt it comes from that) on a fedora 30 server and things work fine. Settings are the same, and while routing tables differ the base idea is the same... Could it be something kernel or virtio related?

holooloo commented 4 years ago

check iptables versions on Centos 7 and 8 must be upper then 1.6.2

ksancheti commented 4 years ago

We are facing the same issue as mentioned by @dmitry-irtegov and @CMajeri.

Environment:

This workaround worked for us -

ethtool -K flannel.1 tx-checksum-ip-generic off

brucedlg commented 4 years ago

It's definitely related to this one: kubernetes/kubernetes#88986 The solution kubernetes/kubernetes#92035 has a good description on the issue. It's the change on iptables rule exposing some existing kernel bug, especially in RHEL7.

Here is another workaround for the issue not requiring turning off chksum offload:

sudo iptables -A OUTPUT -p udp -m udp --dport 8472 -j MARK --set-xmark 0x0

UDP port 8472 is the default port for flannel encapsulating packet. It clears the mark to avoid doing SNAT on the encapsulating packet, thus no double SNAT. This assumes that you use iptables. ipvs should have similar commands.

stale[bot] commented 1 year ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

simondeting commented 4 months ago

检查 Centos 7 和 8 上的 iptables 版本 必须高于 1.6.2

Can you tell me why or give a link? Please.