search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.02k stars 1.34k forks source link

host to service network not working after reboot/join after upgrade from v3.26.3 to v3.27.3 (v3.28.0) ebpf dataplane/vxlan/no kube-proxy/dsr #8867

Closed a-sorokin-sdg closed 2 months ago

a-sorokin-sdg commented 5 months ago

host to service network not working after reboot/join after upgrade from v3.26.3 to v3.27.3 ebpfdataplane/vxlan/no kube-proxy/dsr killing calico-node pod immediately fixing the problem

Expected Behavior

host to service network working well after node reboot/join

Current Behavior

host to kube service network not working after reboot/join node unless you kill calico-node pod it start working after it

Possible Solution

kill calico-node pod on restarted or joined node

kubectl delete pod -n calico-system calico-node-xxxxx

Steps to Reproduce (for bugs)

  1. reboot node
  2. try to connect from the rebooted host to any kube service (kube-dns like example) 
    nslookup example.com {kube-dns svc ip}
  3. fail

Context

Any pods with a host network would fail to start after rebooting/joining node Cluster network works fine

Your Environment

tomastigera commented 5 months ago

killing calico-node pod immediately fixing the problem

What changes after killing the pod. Could you share your routing table before/after?

tomastigera commented 5 months ago

Did you have ctlb disabled before upgrade?

a-sorokin-sdg commented 5 months ago

Did you have ctlb disabled before upgrade?

Nope Can you provide additional parameters which I need to set up? I think it is something with

bpfConnectTimeLoadBalancingEnabled bpfConnectTimeLoadBalancing bpfHostNetworkedNATWithoutCTLB

It's all not set up currently

a-sorokin-sdg commented 5 months ago

killing calico-node pod immediately fixing the problem

What changes after killing the pod. Could you share your routing table before/after?

Routes almost did not change I am checkins via dns resolviong against kube-dns service and it is there after reboot (when it is not working) and after killing pod (when it is working).

10.243.0.10     169.254.1.1     255.255.255.255 UGH   0      0        0 bpfin.cali
tomastigera commented 5 months ago

killing calico-node pod immediately fixing the problem

What changes after killing the pod. Could you share your routing table before/after?

Routes almost did not change I am checkins via dns resolviong against kube-dns service and it is there after reboot (when it is not working) and after killing pod (when it is working).

10.243.0.10     169.254.1.1     255.255.255.255 UGH   0      0        0 bpfin.cali

That route is correct. Since iirc 3.27 we route traffic from host to UDP service via that device by default.

CalicoCon 2024 - Calico eBPF

tomastigera commented 5 months ago

I wonder is some routes caching is in play. Could you dump ip route show cached ?

a-sorokin-sdg commented 5 months ago

I wonder is some routes caching is in play. Could you dump ip route show cached ?

ip route show cached it is empty before and after killing the pod

BTW Apparently, I have a problem only with services pinned in the route table And apparently, they are all UDP services (dns services) There are no tcp services in the route table and they are working fine (tested kube api service via service IP and nginx ingress)

tomastigera commented 5 months ago

Just to confirm, do you see the same problem from host-networked pods/processes or from regular pods as well?

I tried to reproduce the issue, I created a cluster in gcp with kubeadm and installed calico 3.26.4 and upgraded to 3.28 and my DNS did worked just fine.

Would you be able to tcpdump whether your traffic is reaching the service, what kind of packets are exiting from bpfout.cali on your test node?

If your cluster is not a production cluster we could dig deeper with enabling bpf logging to get some more useful logs. Ideally we could sync at calico users slack.

a-sorokin-sdg commented 5 months ago

Just to confirm, do you see the same problem from host-networked pods/processes or from regular pods as well?

Regular pods don't have a problem. They work fine. Tested it to be sure.

So only host-networked pods/processes have problem

a-sorokin-sdg commented 5 months ago

Would you be able to tcpdump whether your traffic is reaching the service, what kind of packets are exiting from bpfout.cali on your test node?

bpfout.cali.pcap.gz

So fresh vm node with single interface joined to cluster I tested with

nslookup openebs-api-rest.openebs.svc.l8s.local. 10.243.0.10

When it's working well after calico-node pod kill bpfout.workin.pcap.gz

tomastigera commented 5 months ago

That doesn't seem to be a problem :arrow_up: Do you see packets returning to the client in both cases? 10.243.0.10 is a local pod or remote?

you could also enable :arrow_down: in default felixconfiguration and provide bpf logs from the node using bpftool prog tracelog > log.txt for the case when it does not work. That should give us good insight.

bpfLogLevel: Debug
bpfLogFilters:
  - all: host 172.24.1.29 and udp port 53
a-sorokin-sdg commented 5 months ago

10.243.0.10 is a local pod or remote?

it's a service IP pods behind that IP are remote hostNetwork -> pod IP has no issue

a-sorokin-sdg commented 5 months ago

bpfLogLevel: Debug bpfLogFilters:

  • all: host 172.24.1.29 and udp port 53

that does not work these changes have been accepted by API

  bpfLogLevel: Debug
  bpfLogFilters:
    all: host 172.24.1.29 and udp port 53

however bpfLogFilters property disappeared from the object so anyway here is a log probably not filtered

bpflog.txt.gz

a-sorokin-sdg commented 5 months ago

BTW I found a repeated error in the tigera operator probably not related to this issue

{"level":"error","ts":"2024-06-12T08:07:07Z","logger":"controller_ippool","msg":"Cannot update an IP pool not owned by the operator","Request.Namespace":"","Request.Name":"periodic-5m0s-reconcile-event","reason":"ResourceValidationError","stacktrace":"github.com/tigera/operator/pkg/controller/status.(*statusManager).SetDegraded\n\t/go/src/github.com/tigera/operator/pkg/controller/status/status.go:356\ngithub.com/tigera/operator/pkg/controller/ippool.(*Reconciler).Reconcile\n\t/go/src/github.com/tigera/operator/pkg/controller/ippool/pool_controller.go:291\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.3/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.3/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.3/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.3/pkg/internal/controller/controller.go:226"}
tomastigera commented 5 months ago

Thanks for the logs, helpful. It seems like the packets from bpfout.cali do not make it to any other device. Perhaps worth verifying with tcpdump. They seem to be eaten by the host network stack. They either have a wrong csum (unlikely that would not get fixed by calico-node restarting) or they get dropped by RPF (could you check the value in /proc/sys/net/ipv4/conf/bpfout.cali/rp_filter) which could be strict and then fixed after the restart. Or it gets dropped by iptables. Do you have a default route? I will give it some more try to reproduce it.

a-sorokin-sdg commented 5 months ago

Before pod killing (after node restart when we have the problem)

cat /proc/sys/net/ipv4/conf/bpfout.cali/rp_filter
1
route ```bash route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.24.1.1 0.0.0.0 UG 100 0 0 eth0 10.243.0.10 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.8.170 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.27.94 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.38.83 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.51.78 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.54.252 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.61.248 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.70.167 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.77.103 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.94.254 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.117.56 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.121.189 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.125.134 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.140.20 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.150.161 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.157.165 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.157.183 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.158.119 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.164.105 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.181.122 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.185.212 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.194.22 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.230.225 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.251.8 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.254.169 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.255.22 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.244.16.0 10.244.16.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.244.32.0 10.244.32.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.244.48.0 0.0.0.0 255.255.240.0 U 0 0 0 * 10.244.48.6 0.0.0.0 255.255.255.255 UH 0 0 0 califee8cfb24e3 10.244.48.7 0.0.0.0 255.255.255.255 UH 0 0 0 calie89ffdb4633 10.244.48.8 0.0.0.0 255.255.255.255 UH 0 0 0 cali193e57c628e 10.244.48.9 0.0.0.0 255.255.255.255 UH 0 0 0 calid31be247766 10.244.192.0 10.244.192.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.245.80.0 10.245.80.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.246.96.0 10.246.96.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.247.80.0 10.247.80.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.247.112.0 10.247.112.0 255.255.240.0 UG 0 0 0 vxlan.calico 169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 bpfin.cali 172.24.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0 ```
iptables ```bash iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:zkuE8qdwsVpH6Kd2 */ /* Accept packets from flows that pre-date BPF. */ mark match 0x5000000/0x5000000 ctstate RELATED,ESTABLISHED DROP all -- anywhere anywhere /* cali:XQL0mC-L6wldZdgN */ /* Drop packets from unknown flows. */ mark match 0x5000000/0x5000000 ACCEPT all -- anywhere anywhere /* cali:pbFdTFCLcV-MVLSS */ mark match 0x1000000/0x1000000 DROP all -- anywhere anywhere /* cali:u_TyW7ph8QsYnThE */ mark match ! 0x1000000/0x1000000 KUBE-FIREWALL all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:umcmOn0WnTNOKJrp */ /* Pre-approved by BPF programs. */ mark match 0x3000000/0x3000000 DROP all -- anywhere anywhere /* cali:NnQ109Z-tVFkJGc1 */ /* From workload without BPF seen mark */ mark match ! 0x1000000/0x1000000 MARK all -- anywhere anywhere /* cali:YmI_zfAgHIHbINEV */ /* Mark pre-established flows. */ ctstate RELATED,ESTABLISHED MARK or 0x8000000 cali-to-wl-dispatch all -- anywhere anywhere /* cali:-EFgmtwMJVO64q9s */ /* To workload, check workload is known. */ ACCEPT all -- anywhere anywhere /* cali:wP1i1sEU71uRzM5d */ /* To workload, mark has already been verified. */ ACCEPT all -- anywhere anywhere /* cali:3t5V_2xe4DFVlHBq */ /* From */ /* bpfout.cali */ /* device, mark verified, accept. */ Chain OUTPUT (policy ACCEPT) target prot opt source destination MARK all -- anywhere anywhere /* cali:CD7jZCSqPP_KjGsd */ /* Mark pre-established flows. */ ctstate RELATED,ESTABLISHED MARK or 0x8000000 KUBE-FIREWALL all -- anywhere anywhere Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT Chain KUBE-KUBELET-CANARY (0 references) target prot opt source destination Chain cali-to-wl-dispatch (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:cEwZ48PLVj36YM8T */ ACCEPT all -- anywhere anywhere /* cali:Geyg9JmnnDNPlLHX */ ACCEPT all -- anywhere anywhere /* cali:ICRLMI_1Qq8A0HWR */ ACCEPT all -- anywhere anywhere /* cali:bhG4cOJs_TnufY0G */ DROP all -- anywhere anywhere /* cali:k6ZOE-XDbClrqbFe */ /* Unknown interface */ ```

After pod killing (when it is working well)

cat /proc/sys/net/ipv4/conf/bpfout.cali/rp_filter
0
route ```bash route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 0.0.0.0 172.24.1.1 0.0.0.0 UG 100 0 0 eth0 10.243.0.10 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.8.170 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.27.94 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.38.83 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.51.78 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.54.252 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.61.248 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.70.167 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.77.103 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.94.254 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.117.56 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.121.189 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.125.134 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.140.20 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.150.161 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.157.165 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.157.183 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.158.119 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.164.105 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.181.122 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.185.212 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.194.22 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.230.225 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.251.8 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.254.169 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.243.255.22 169.254.1.1 255.255.255.255 UGH 0 0 0 bpfin.cali 10.244.16.0 10.244.16.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.244.32.0 10.244.32.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.244.48.0 0.0.0.0 255.255.240.0 U 0 0 0 * 10.244.48.6 0.0.0.0 255.255.255.255 UH 0 0 0 califee8cfb24e3 10.244.48.7 0.0.0.0 255.255.255.255 UH 0 0 0 calie89ffdb4633 10.244.48.8 0.0.0.0 255.255.255.255 UH 0 0 0 cali193e57c628e 10.244.48.9 0.0.0.0 255.255.255.255 UH 0 0 0 calid31be247766 10.244.192.0 10.244.192.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.245.80.0 10.245.80.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.246.96.0 10.246.96.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.247.80.0 10.247.80.0 255.255.240.0 UG 0 0 0 vxlan.calico 10.247.112.0 10.247.112.0 255.255.240.0 UG 0 0 0 vxlan.calico 169.254.1.1 0.0.0.0 255.255.255.255 UH 0 0 0 bpfin.cali 172.24.1.0 0.0.0.0 255.255.255.0 U 100 0 0 eth0 ```
iptables ```bash iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:zkuE8qdwsVpH6Kd2 */ /* Accept packets from flows that pre-date BPF. */ mark match 0x5000000/0x5000000 ctstate RELATED,ESTABLISHED DROP all -- anywhere anywhere /* cali:XQL0mC-L6wldZdgN */ /* Drop packets from unknown flows. */ mark match 0x5000000/0x5000000 ACCEPT all -- anywhere anywhere /* cali:pbFdTFCLcV-MVLSS */ mark match 0x1000000/0x1000000 DROP all -- anywhere anywhere /* cali:u_TyW7ph8QsYnThE */ mark match ! 0x1000000/0x1000000 KUBE-FIREWALL all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:umcmOn0WnTNOKJrp */ /* Pre-approved by BPF programs. */ mark match 0x3000000/0x3000000 DROP all -- anywhere anywhere /* cali:NnQ109Z-tVFkJGc1 */ /* From workload without BPF seen mark */ mark match ! 0x1000000/0x1000000 MARK all -- anywhere anywhere /* cali:YmI_zfAgHIHbINEV */ /* Mark pre-established flows. */ ctstate RELATED,ESTABLISHED MARK or 0x8000000 cali-to-wl-dispatch all -- anywhere anywhere /* cali:-EFgmtwMJVO64q9s */ /* To workload, check workload is known. */ ACCEPT all -- anywhere anywhere /* cali:wP1i1sEU71uRzM5d */ /* To workload, mark has already been verified. */ ACCEPT all -- anywhere anywhere /* cali:3t5V_2xe4DFVlHBq */ /* From */ /* bpfout.cali */ /* device, mark verified, accept. */ Chain OUTPUT (policy ACCEPT) target prot opt source destination MARK all -- anywhere anywhere /* cali:CD7jZCSqPP_KjGsd */ /* Mark pre-established flows. */ ctstate RELATED,ESTABLISHED MARK or 0x8000000 KUBE-FIREWALL all -- anywhere anywhere Chain KUBE-FIREWALL (2 references) target prot opt source destination DROP all -- !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT Chain KUBE-KUBELET-CANARY (0 references) target prot opt source destination Chain cali-to-wl-dispatch (1 references) target prot opt source destination ACCEPT all -- anywhere anywhere /* cali:cEwZ48PLVj36YM8T */ ACCEPT all -- anywhere anywhere /* cali:Geyg9JmnnDNPlLHX */ ACCEPT all -- anywhere anywhere /* cali:ICRLMI_1Qq8A0HWR */ ACCEPT all -- anywhere anywhere /* cali:bhG4cOJs_TnufY0G */ DROP all -- anywhere anywhere /* cali:k6ZOE-XDbClrqbFe */ /* Unknown interface */ ```
tomastigera commented 5 months ago

cat /proc/sys/net/ipv4/conf/bpfout.cali/rp_filter -> 1 vs cat /proc/sys/net/ipv4/conf/bpfout.cali/rp_filter -> 0

That is the problem. Something sets it to 1(strict) and when calico-node restarts, is sets it back to 0. The something is probably your systemd which applies configuration when a new device is added. Seems like the issue is present with systemd 245+ What is your linux distro (which I should have asked a while ago)?

a-sorokin-sdg commented 5 months ago

What is your linux distro (which I should have asked a while ago)?

# uname -r
5.14.0-452.el9.x86_64
# cat /etc/os-release
NAME="CentOS Stream"
VERSION="9"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="9"
PLATFORM_ID="platform:el9"
PRETTY_NAME="CentOS Stream 9"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:centos:centos:9"
HOME_URL="https://centos.org/"
BUG_REPORT_URL="https://issues.redhat.com/"
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"
a-sorokin-sdg commented 5 months ago

he something is probably your systemd

in sysctl setting it is set as

net.ipv4.conf.all.rp_filter=0
cat /etc/sysctl.conf 

fs.inotify.max_user_instances=1048576
fs.inotify.max_user_watches=1048576
fs.inotify.max_queued_events=16384
fs.aio-max-nr=1048576
vm.max_map_count=262144
net.ipv4.ip_nonlocal_bind=1
net.ipv4.ip_forward=1
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv4.neigh.default.gc_thresh1=8192
net.ipv4.neigh.default.gc_thresh2=12228
net.ipv4.neigh.default.gc_thresh3=24456
net.core.somaxconn=65535
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.all.accept_local=1
kernel.panic=30
kernel.panic_on_oops=1
vm.overcommit_memory=2
vm.panic_on_oom=0
tomastigera commented 2 months ago

@a-sorokin-sdg do you still see the issue? Have you figured what is changing the rpf? Closing now, but feel free to reopen if you have any new info.

a-sorokin-sdg commented 2 months ago

Yes, still have the issue. I have tried to catch who change it via audit without success.