search

antrea-io / antrea

Kubernetes networking based on Open vSwitch
https://antrea.io
Apache License 2.0
1.67k stars 371 forks source link

ip6tables agent error on dual-stack with multicast enabled #6113

Closed lezruk closed 8 months ago

lezruk commented 8 months ago

Describe the bug

Agent failing to initialize node network becuase of ip6tables failure in agent with enable dual-stack and multicast

To Reproduce Enable cluster dual stack Enable Antrea multicast support

Expected Traffic rule to be applied to IPv4 only

Actual behavior ip6tables command error Agent log:

antrea-agent E0315 21:03:18.404443       1 iptables.go:358] "Failed to execute iptables command" err="exit status 2" iptablesCmd="ip6tables-restore" stdin=<                                                     
antrea-agent     *raw                                                                                                                                                                                            
antrea-agent     :ANTREA-PREROUTING - [0:0]                                                                                                                                                                      
antrea-agent     :ANTREA-OUTPUT - [0:0]                                                                                                                                                                          
antrea-agent     -A ANTREA-PREROUTING -m comment --comment "Antrea: do not track incoming encapsulation packets" -m udp -p udp --dport 6081 -m addrtype --dst-type LOCAL -j NOTRACK                              
antrea-agent     -A ANTREA-OUTPUT -m comment --comment "Antrea: do not track outgoing encapsulation packets" -m udp -p udp --dport 6081 -m addrtype --src-type LOCAL -j NOTRACK                                  
antrea-agent     -A ANTREA-PREROUTING -m comment --comment "Antrea: drop Pod multicast traffic forwarded via underlay network" -m set --match-set CLUSTER-NODE-IP6 src -d 224.0.0.0/4 -j DROP                    
antrea-agent     COMMIT                                                                                                                                                                                          
antrea-agent     *mangle                                                                                                                                                                                         
antrea-agent     :ANTREA-MANGLE - [0:0]                                                                                                                                                                          
antrea-agent     :ANTREA-OUTPUT - [0:0]                                                                                                                                                                          
antrea-agent     -A ANTREA-OUTPUT -m comment --comment "Antrea: mark LOCAL output packets" -m addrtype --src-type LOCAL -o antrea-gw0 -j MARK --or-mark 0x80000000                                               
antrea-agent     COMMIT                                                                                                                                                                                          
antrea-agent     *filter                                                                                                                                                                                         
antrea-agent     :ANTREA-FORWARD - [0:0]                                                                                                                                                                         
antrea-agent     -A ANTREA-FORWARD -m comment --comment "Antrea: accept packets from local Pods" -i antrea-gw0 -j ACCEPT                                                                                         
antrea-agent     -A ANTREA-FORWARD -m comment --comment "Antrea: accept packets to local Pods" -o antrea-gw0 -j ACCEPT                                                                                           
antrea-agent     COMMIT                                                                                                                                                                                          
antrea-agent     *nat                                                                                                                                                                                            
antrea-agent     :ANTREA-PREROUTING - [0:0]                                                                                                                                                                      
antrea-agent     -A ANTREA-PREROUTING -m comment --comment "Antrea: DNAT external to NodePort packets" -m set --match-set ANTREA-NODEPORT-IP6 dst,dst -j DNAT --to-destination fc01::aabb:ccdd:eefe              
antrea-agent     :ANTREA-OUTPUT - [0:0]                                                                                                                                                                          
antrea-agent     -A ANTREA-OUTPUT -m comment --comment "Antrea: DNAT local to NodePort packets" -m set --match-set ANTREA-NODEPORT-IP6 dst,dst -j DNAT --to-destination fc01::aabb:ccdd:eefe                     
antrea-agent     :ANTREA-POSTROUTING - [0:0]                                                                                                                                                                     
antrea-agent     -A ANTREA-POSTROUTING -m comment --comment "Antrea: masquerade Pod to external packets" -s fd00:cafe:42::/64 -m set ! --match-set ANTREA-POD-IP6 dst ! -o antrea-gw0 -j MASQUERADE              
antrea-agent     -A ANTREA-POSTROUTING -m comment --comment "Antrea: masquerade LOCAL traffic" -o antrea-gw0 -m addrtype ! --src-type LOCAL --limit-iface-out -m addrtype --src-type LOCAL -j MASQUERADE --randomantrea-agent     -A ANTREA-POSTROUTING -m comment --comment "Antrea: masquerade OVS virtual source IP" -s fc01::aabb:ccdd:eeff -j MASQUERADE                                                                     
antrea-agent     COMMIT                                                                                                                                                                                          
antrea-agent  > stderr= <                                                                                                                                                                                        
antrea-agent     ip6tables-restore v1.8.7 (nf_tables): host/network `224.0.0.0' not found                                                                                                                        
antrea-agent     Error occurred at line: 6                                                                                                                                                                       
antrea-agent     Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.                                                                                                                  
antrea-agent  >                                                                                                                                                                                                  
antrea-agent E0315 21:03:18.404510       1 route_linux.go:198] Failed to initialize iptables: error executing ip6tables-restore: exit status 2 - will retry in 2s

Versions:

k3s:v1.29.2+k3s1

Linux vk3scontrolt0 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

Default Debian 12.5

vk3scontrolt0 ~ # apt show openvswitch-switch
Package: openvswitch-switch
Version: 3.1.0-2+deb12u1
Priority: optional
Section: net
Source: openvswitch
Maintainer: Debian OpenStack <team+openstack@tracker.debian.org>
Installed-Size: 8 391 kB
Pre-Depends: init-system-helpers (>= 1.54~)
Depends: kmod | module-init-tools, netbase, openvswitch-common (= 3.1.0-2+deb12u1), procps, python3-netifaces, python3-openvswitch (>= 3.1.0-2+deb12u1), uuid-runtime, python3:any, libbpf1 (>= 1:0.7.0), libc6 (>= 2.34), libcap-ng0 (>= 0.7.9), libnuma1 (>= 2.0.11), libssl3 (>= 3.0.0), libunbound8 (>= 1.8.0), libxdp1 (>= 1.2.9~)
Breaks: openvswitch-common (<< 2.17~)
Replaces: openvswitch-common (<< 2.17~)
Homepage: http://openvswitch.org/
Download-Size: 1 693 kB
APT-Sources: http://security.debian.org/debian-security bookworm-security/main amd64 Packages

Additional context

antoninbas commented 8 months ago

Thanks for the report. However, we do not support Multicast with IPv6 (see https://github.com/antrea-io/antrea/blob/main/docs/multicast-guide.md#limitations). @wenyingd @ceclinux could we fail early if Multicast is enabled in an IPv6 cluster?

We should probably also open an issue to discuss adding this support. I don't know if anything specific was identified as a technical blocker for implementing Multicast with IPv6.

wenyingd commented 8 months ago

We should probably also open an issue to discuss adding this support. I don't know if anything specific was identified as a technical blocker for implementing Multicast with IPv6.

The design for Multicast is generic for both IPv4 and IPv6. The major difference with IPv6 is we need to use MLD protocol as an alternative for IGMP which only works for IPv4. From the openflow library (libOpenflow), MLD packet format was already added, but we didn't add the corresponding invocation in antrea logic. And another reason for not support it in IPv6 is we didn't receive a user request on it yet, so it is processed with a lower priority until now.

tnqn commented 8 months ago

Before Multicast can be supported with IPv6, does it make sense to make the code self-adaptive that it would enable Multicast for IPv4 and skip IPv6 for a dual-stack cluster? I feel there may be use cases that the Multicast is only needed for IPv4 even though the whole cluster is dual-stack. The validation could be:

@lezruk do you need multicast to work for IPv6, or you only need it with IPv4 in a dual-stack cluster?

lezruk commented 8 months ago

Hi @tnqn thank you. I understand IPv6 multicast is not there yet , here the case is to have IPv4 multicast in dual-stack configuration for IPv4 only (which is failling becuase IPv4 multicast prerouting rule is tried to be applied for IPv6). This is exact problem here

-A ANTREA-PREROUTING -m comment --comment "Antrea: drop Pod multicast traffic forwarded via underlay network" -m set --match-set **CLUSTER-NODE-IP6** src -d **224.0.0.0/4** -j DROP

@lezruk do you need multicast to work for IPv6, or you only need it with IPv4 in a dual-stack cluster?

To be more precise answering your question: at the moment to have only IPv4 multicast working in dual-stack.

wenyingd commented 8 months ago

@antoninbas @tnqn Having looked through the code, it is difficult to check the Multicast feature gate status along with the IPv6 cluster in agent option validation stage (https://github.com/antrea-io/antrea/blob/main/cmd/antrea-agent/options.go#L141), since antrea-agent didn't get the IPv4/IPv6/dual-stack settings at that time. So if we plan to check and quit as early as we can, the earliest chance is after the agent Initialize in run (https://github.com/antrea-io/antrea/blob/main/cmd/antrea-agent/agent.go#L303). But it looks strange to add such check in agent.run. A thought in my mind is to add the check inside agentInitializer.initNodeLocalConfig (https://github.com/antrea-io/antrea/blob/main/pkg/agent/agent.go#L1302) . What is your thought?

antoninbas commented 8 months ago

Maybe we could do it inside the call to mcastController.Initialize. This function is only called if Multicast is enabled, and it should know if the cluster is IPv4 / IPv6 / dual-stack. AFAIK, the agentInitializer is not currently aware of whether Multicast is enabled, so it would require providing that information to it?