search

squat / kilo

Kilo is a multi-cloud network overlay built on WireGuard and designed for Kubernetes (k8s + wg = kg)
https://kilo.squat.ai
Apache License 2.0
2.02k stars 122 forks source link

High Cpu Usage #135

Closed Ehco1996 closed 3 years ago

Ehco1996 commented 3 years ago

today i restart one k3s node , and restart the kilo pod

after that kilo pod use a lot of cpu :

截屏2021-03-12 上午8 13 45

the kilo log show that "kilo-gnxvg kilo {"caller":"iptables.go:308","component":"iptables","level":"info","msg":"applying 7 iptables rules","ts":"2021-03-12T00:15:23.452988995Z"}"

i think maybe there is an infinite loop in the code which always flush the iptables rule ?

Ehco1996 commented 3 years ago

updated：

i downgrade kilo to three month ago version, erverything is ok

also i found that the leatest kilo image was updated ten minutes ago

so maybe there is some thing wrong with the new code ?

squat commented 3 years ago

Hi @Ehco1996, thanks a lot for reporting this. Yes, the new code from https://github.com/squat/kilo/commit/acfd0bbaec239f0a3ec6625828444ac80930afcf is known to be fragile because the iptables controller is comparing the text of iptables rules literally rather than semantically. We have a plan to replace this with an iptables rules parser very soon to make this more robust. Ironically, this change was made to reduce the CPU utilization of Kilo in https://github.com/squat/kilo/issues/113.

In order to understand why the iptables controller is failing for you, could you share the output of sudo iptables-save | grep -i KILO?

Thanks!

Ehco1996 commented 3 years ago

@squat thanks for quick reply.

after i downgrade kilo to squat/kilo:amd64-a789003a582f09286331aaa4d4d285f1ef4bd223, kilo works good

logs :

{"caller":"mesh.go:513","component":"kilo","level":"info","msg":"WireGuard configurations are different","ts":"2021-03-13T00:56:48.420610962Z"}

iptables rules:

:KILO-IPIP - [0:0]
-A INPUT -p ipv4 -m comment --comment "Kilo: jump to IPIP chain" -j KILO-IPIP
-A INPUT -p ipv4 -m comment --comment "Kilo: reject other IPIP traffic" -j DROP
:KILO-NAT - [0:0]
-A POSTROUTING -s 10.42.2.0/24 -m comment --comment "Kilo: jump to NAT chain" -j KILO-NAT
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.4.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.24.238/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.10.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.188.42/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.6.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.34.8/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.9.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.32.236/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.5/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.7.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.33.147/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.5/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.6/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.8.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.33.155/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.6/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.7/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.0.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.7/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.8/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.5.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.8/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.9/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.3.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.17/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.9/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.10/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.1.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.10/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.2.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -m comment --comment "Kilo: NAT remaining packets" -j MASQUERADE

to reproduce the high cpu usage ,i upgrade kilo to the latest ver

iprtables rules:

:KILO-IPIP - [0:0]
-A INPUT -p ipv4 -m comment --comment "Kilo: jump to IPIP chain" -j KILO-IPIP
-A INPUT -p ipv4 -m comment --comment "Kilo: reject other IPIP traffic" -j DROP
:KILO-NAT - [0:0]
-A POSTROUTING -s 10.42.4.0/24 -m comment --comment "Kilo: jump to KILO-NAT chain" -j KILO-NAT
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.4.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.24.238/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.1/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.10.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.188.42/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.2/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.6.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.34.8/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.3/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.9.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.32.236/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.5/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.7.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.33.147/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.5/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.6/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.8.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 172.17.33.155/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.6/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.7/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.0.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.7/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.8/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.5.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.4/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.8/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.9/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.3.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.0.0.17/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.9/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.10/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.1.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.10/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for WireGuared IPs" -j RETURN
-A KILO-NAT -d 10.42.2.0/24 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -d 10.4.0.11/32 -m comment --comment "Kilo: do not NAT packets destined for known IPs" -j RETURN
-A KILO-NAT -m comment --comment "Kilo: NAT remaining packets" -j MASQUERADE
squat commented 3 years ago

Thanks for the help! From your iptables-save dump it seems clear to me that the problem is that #116 didn't fix the IPIP rules in https://github.com/squat/kilo/blob/main/pkg/encapsulation/ipip.go#L68-L83. I'll sort this out ASAP

squat commented 3 years ago

@Ehco1996 ok, we just merged a fix for this bug! plz try out the latest Kilo and re-open the issue if you still have problems. Thanks for your collaboration :)