issue with container traffic across nodes after reboot

[jr200]

I've installed k3s on top of wireguard via flannel by following the netmaker tutorial on youtube. I experienced an issue running the pingtest after a reboot.

Steps to reproduce:

1. deploy pingtest to nodes A,B,C,D (with resp. subnets 10.42.0.1/24, 10.42.1.1/24, 10.42.2.1/24, 10.42.3.1/24)
2. kubectl -n pingtest exec -it {pod-B} -- ping {IP-A,C,D} => works!!
3. kubectl -n pingtest exec -it {pod-A,C,D} -- ping {IP-B} => works!!
4. reboot B
5. kubectl get nodes -o wide => shows B is Ready
6. kubectl -n pingtest exec -it {pod-B} -- ping {IP-A,C,D} => doesnt work!!
7. kubectl -n pingtest exec -it {pod-A,C,D} -- ping {IP-B} => doesnt work!!
8. systemctl restart k3s-agent.service on node B
9. kubectl -n pingtest exec -it {pod-B} -- ping {IP-A,C,D} => works!!
10. kubectl -n pingtest exec -it {pod-A,C,D} -- ping {IP-B} => works!!

I wonder if anyone else is experiencing this issue, or if its entirely a problem with my setup.

I'm using ubuntu server 21.04, k3s v1.21.1+k3s1.

[afeiszli]

@jr200 following up with additional data based on discord conversation: This is due to rotating public keys. If a node reboots, it gets a new public key. It is good to rotate keys for security reasons, but also important to make sure connections work afterwards. I believe what you found was the connection did work after a little while (it takes about 1-2 minutes for the new public key to propagate to other nodes). Is this correct? If so I would not call this one a bug.

However, there is a separate issue with Ingress Gateways, because public key rotation will break the client configuration. For this we should have a separate issue for maintaining a static public key on ingress gateways (unless manually rotated).

[afeiszli]

@jr200 this has been fixed in the new release. I will close once 0.7 is official (slated for 8/11/21).

[afeiszli]

Fixed in 0.7