rbrtbnfgl
commented
8 months ago Hi thanks for reporting this. Checking from the code the public-ip is used only to select the interface and what you are saying is right. We can rework the code to force to use the same defined IP in case of multiple IPs on that interface.
Edit: I think I misunderstood what you wrote. The annotation public-ip is not used by flannel to select the IP but it's configured by the CNI itself. The flannel configuration is specified only by the cli options on the current implementation and not from the annotations.
Edit 2: you can configure the helm chart values with RKE2 without editing the chart itself.
dvgt
dvergotes
Expected Behavior
The node annotation
flannel.alpha.coreos.com/public-ipv6orflannel.alpha.coreos.com/public-ipv6-overwrite(if set), should have the same impact as setting the--public-ipv6option of theflanneldbinary.Current Behavior
--public-ipv6cli option, the external address used by flannel seems to always be the first address of the interface which also has the address from the annotations.--public-ipv6cli option, the external address used by flannel is always the address given by the cli option.Possible Solution
Steps to Reproduce (for bugs)
Example config for a master node (/etc/rancher/rke2/config.yaml):
Example config for a worker node (/etc/rancher/rke2/config.yaml)
On the worker node, add an extra IPv6 address to the interface that is used for inter-pod traffic
ip a add 2001:x:x:x:x:x:x:229/64 dev ens192Delete the canal pod that's running on the worker node
kubectl delete pod rke2-canal-... -n kube-systemObserve that the flannel logs of the worker node mention the wrong external address (
2001:x:x:x:x:x:x:229iso2001:x:x:x:x:x:x:105)I1004 16:30:13.143547 1 main.go:543] Found network config - Backend type: vxlan I1004 16:30:13.143620 1 match.go:206] Determining IP address of default interface I1004 16:30:13.146384 1 match.go:259] Using interface with name ens192 and address 10.50.147.105 I1004 16:30:13.146440 1 match.go:262] Using interface with name ens192 and v6 address 2001:x:x:x:x:x:x:229 I1004 16:30:13.146463 1 match.go:281] Defaulting external address to interface address (10.50.147.105) I1004 16:30:13.146486 1 match.go:294] Defaulting external v6 address to interface address (2001:x:x:x:x:x:x:229)
NODE1="master-node" NODE2="worker-node" for NODE in ${NODE1} ${NODE2}; do \ kubectl run --restart=Never --overrides="{ \"spec\": { \"nodeSelector\": { \"kubernetes.io/hostname\": \"${NODE}\" } } }" --image=docker.io/library/busybox:1.28 busybox-${NODE} -- sh -c 'sleep 36000'; \ done sleep 10 IPV6POD2=$(kubectl get pods busybox-${NODE2} -o custom-columns=IP:".status.podIPs[1].ip" --no-headers); echo ${IPV6POD2} kubectl exec -it busybox-${NODE1} -- ping -6 ${IPV6POD2}
tcpdump -i ens192 -pnnev ip6 16:37:52.833187 aa:bb:cc:dd:ee:01 > aa:bb:cc:dd:ee:02, ethertype IPv6 (0x86dd), length 188: (hlim 64, next-header UDP (17) payload length: 134) 2001:x:x:x:x:x:x:218.59599 > 2001:x:x:x:x:x:x:105.8472: [bad udp cksum 0x187f -> 0xcd86!] OTV, flags [I] (0x08), overlay 0, instance 1 86:8a:9a:80:a1:c5 > f6:18:3e:eb:4d:18, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xda0ae, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:x:x:y::301f > 2001:x:x:y::4023: [icmp6 sum ok] ICMP6, echo request, seq 7 16:37:52.833575 aa:bb:cc:dd:ee:02 > aa:bb:cc:dd:ee:01, ethertype IPv6 (0x86dd), length 188: (hlim 64, next-header UDP (17) payload length: 134) 2001:x:x:x:x:x:x:229.54910 > 2001:x:x:x:x:x:x:218.8472: [udp sum ok] OTV, flags [I] (0x08), overlay 0, instance 1 f6:18:3e:eb:4d:18 > 86:8a:9a:80:a1:c5, ethertype IPv6 (0x86dd), length 118: (flowlabel 0xd15b5, hlim 63, next-header ICMPv6 (58) payload length: 64) 2001:x:x:y::4023 > 2001:x:x:y::301f: [icmp6 sum ok] ICMP6, echo reply, seq 7