search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
6.01k stars 1.34k forks source link

Log spam: felix/int_dataplane.go 1800: Failed to synchronize routing table, will retry... #4944

Closed lwr20 closed 5 months ago

lwr20 commented 3 years ago

My calico-node logs are being filled with this sort of thing:

2021-09-17 18:19:59.003 [WARNING][54] felix/int_dataplane.go 1800: Failed to synchronize routing table, will retry...
2021-09-17 18:19:59.103 [INFO][54] felix/route_table.go 448: Trying to connect to netlink
2021-09-17 18:19:59.103 [INFO][54] felix/route_table.go 944: Deleting from expected targets cidr=192.168.180.0/26 ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.104 [INFO][54] felix/route_table.go 950: No pending target update, adding back in as an update cidr=192.168.180.0/26 ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.104 [INFO][54] felix/route_table.go 1142: No interface on route. error=netlink update operation failed ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.104 [INFO][54] felix/route_table.go 448: Trying to connect to netlink
2021-09-17 18:19:59.104 [INFO][54] felix/route_table.go 944: Deleting from expected targets cidr=192.168.180.0/26 ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.104 [INFO][54] felix/route_table.go 950: No pending target update, adding back in as an update cidr=192.168.180.0/26 ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.105 [WARNING][54] felix/route_table.go 701: Failed to add route error=file exists ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.105 [INFO][54] felix/route_table.go 1142: No interface on route. error=netlink update operation failed ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.105 [WARNING][54] felix/route_table.go 586: Failed to sync routes to interface even after retries. Leaving it dirty, requiring a full sync. ifaceName="*NoOIF*" ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.105 [WARNING][54] felix/route_table.go 599: Some interfaces still out-of sync. ifaceRegex="" ipVersion=0x4 route_table="vxlan_blackhole"
2021-09-17 18:19:59.106 [WARNING][54] felix/int_dataplane.go 1800: Failed to synchronize routing table, will retry...
2021-09-17 18:19:59.206 [INFO][54] felix/route_table.go 448: Trying to connect to netlink

Expected Behavior

Logs shouldn't be full of spam.

Current Behavior

Logs being spammed :)

Possible Solution

Steps to Reproduce (for bugs)

kubeadm installed cluster on GCP. k8s 1.20. Calico 3.20.1. Installed with:

kubectl apply -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl apply -f https://docs.projectcalico.org/manifests/custom-resources.yaml

Context

I can't see any other calico-node logs, because I'm being spammed by those every 100ms.

Your Environment

Rathgore commented 3 years ago

Running into this as well with Calico v3.20.1. Dropping back to v3.19.3 fixes the issue. This on an AWS EKS cluster running Kubernetes v1.21.2.

ceastman-r7 commented 2 years ago

I have similar logs: calico-node-7w96s calico-node 2022-11-04 14:49:13.328 [WARNING][60] felix/route_table.go 735: Failed to delete route error=no such process ifaceName="eni5432d136d5c" ifaceRegex="^eni." ipVersion=0x4 tableIndex=0 calico-node-7w96s calico-node 2022-11-04 14:49:53.189 [INFO][60] felix/route_table.go 1200: Failed to access interface because it doesn't exist. error=Link not found ifaceName="enibbdbe78f3d9" ifaceRegex="^eni." ipVersion=0x4 tableIndex=0

iAlexBLR commented 1 year ago

Had the same issue on AWS EKS v1.23 and had to downgrade to v3.19.4 like it was suggested above. Calico version: v3.20.1

Git4Mark commented 1 year ago

Had the same issue on kubernetes v1.24.4. Calico version: v3.24.0

lwr20 commented 1 year ago

@Git4Mark can you post the errors you're seeing please, would be good to know the current line numbers of the offending logs.

Git4Mark commented 1 year ago

@Git4Mark can you post the errors you're seeing please, would be good to know the current line numbers of the offending logs. i have 5 nodes,just one can't startup

2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.30.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.30.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.59.192/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.59.192/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.241.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.241.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.235.0/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.100 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.235.0/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.30.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.30.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.59.192/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.59.192/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.241.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.241.64/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 945: Deleting from expected targets cidr=10.51.235.0/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [INFO][99] felix/route_table.go 951: No pending target update, adding back in as an update cidr=10.51.235.0/26 ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 752: Failed to add route error=file exists ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 route={Ifindex: 163 Dst: 10.51.30.64/26 Src: <nil> Gw: 10.51.30.75 Flags: [onlink] Table: 0 Realm: 0} tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 752: Failed to add route error=file exists ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 route={Ifindex: 163 Dst: 10.51.59.192/26 Src: <nil> Gw: 10.51.59.197 Flags: [onlink] Table: 0 Realm: 0} tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 752: Failed to add route error=file exists ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 route={Ifindex: 163 Dst: 10.51.241.64/26 Src: <nil> Gw: 10.51.241.65 Flags: [onlink] Table: 0 Realm: 0} tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 752: Failed to add route error=file exists ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 route={Ifindex: 163 Dst: 10.51.235.0/26 Src: <nil> Gw: 10.51.235.3 Flags: [onlink] Table: 0 Realm: 0} tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 1222: Failed to access interface but it appears to be up error=netlink update operation failed ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 link=&netlink.Vxlan{LinkAttrs:netlink.LinkAttrs{Index:163, MTU:1410, TxQLen:0, Name:"vxlan.calico", HardwareAddr:net.HardwareAddr{0x66, 0x8c, 0x33, 0x86, 0x44, 0xce}, Flags:0x13, RawFlags:0x11043, ParentIndex:0, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0xc000f3be00), Promisc:0, Allmulti:0, Multi:1, Xdp:(*netlink.LinkXdp)(0xc0011897d0), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x0, PhysSwitchID:0, NetNsID:-1, NumTxQueues:1, NumRxQueues:1, GSOMaxSize:0x10000, GSOMaxSegs:0xffff, GROMaxSize:0x0, Vfs:[]netlink.VfInfo(nil), Group:0x0, Slave:netlink.LinkSlave(nil)}, VxlanId:4096, VtepDevIndex:161, SrcAddr:net.IP{0xa, 0x4, 0x0, 0x1}, Group:net.IP(nil), TTL:0, TOS:0, Learning:false, Proxy:false, RSC:false, L2miss:false, L3miss:false, UDPCSum:true, UDP6ZeroCSumTx:false, UDP6ZeroCSumRx:false, NoAge:false, GBP:false, FlowBased:false, Age:300, Limit:0, Port:4789, PortLow:0, PortHigh:0} tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 635: Failed to sync routes to interface even after retries. Leaving it dirty, requiring a full sync. ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/route_table.go 648: Some interfaces still out-of sync. ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
2023-06-06 02:27:47.101 [WARNING][99] felix/int_dataplane.go 2008: Failed to synchronize routing table, will retry...
lwr20 commented 1 year ago

i have 5 nodes,just one can't startup

@Git4Mark This Issue is for log spam only, if you have a node which can't start up, that suggests there's a different problem.

Can you either raise as a separate issue or join us in slack https://slack.projectcalico.org for further assistance please?