search

multipath-tcp / mptcp

⚠️⚠️⚠️ Deprecated 🚫 Out-of-tree Linux Kernel implementation of MultiPath TCP. 👉 Use https://github.com/multipath-tcp/mptcp_net-next repo instead ⚠️⚠️⚠️
https://github.com/multipath-tcp/mptcp_net-next
Other
890 stars 336 forks source link

kern.err kernel:__mptcp_init6_subsockets: MPTCP subsocket connect() failed, error -13 #499

Closed kevinh-csalabs closed 1 year ago

kevinh-csalabs commented 2 years ago

My logs are getting flooded with the message kern.err kernel:__mptcp_init6_subsockets: MPTCP subsocket connect() failed, error -13 Which clearly relates to IPv6 but there doesn't seem to be much else to go on to determine what error 13 is.

matttbe commented 2 years ago

Hi @kevinh-csalabs

Thank you for the bug report but we will need more detail about your setup to understand what's wrong:

Error -13 is EACCES: Permission denied. So maybe something not allowing the kernel to create subflows (SELinux? etc.)

kevinh-csalabs commented 2 years ago
  • which kernel version are you using?
Linux OpenMPTCProuter 5.4.194 #0 SMP Tue May 17 22:11:28 2022 aarch64 GNU/Linux
  • how did you install it?

Packaged as part of OpenMPTCProuter distribution

  • on which Linux distribution?

OpenMPTCProuter

  • on which kind of device?

Raspberry Pi 4

  • what's your network setup?

3 wan connections (lte, 5g, starlink) configured through tunnel to vps

  • which IP routes/rules did you configure?
default         10.255.254.1    0.0.0.0         UG    0      0        0 tun0
default         192.168.12.1    0.0.0.0         UG    1      0        0 eth0.3
default         100.64.0.1      0.0.0.0         UG    10     0        0 eth0.2
default         192.168.12.1    0.0.0.0         UG    12     0        0 eth0.3
default         10.191.199.1    0.0.0.0         UG    14     0        0 eth0.4
default         10.255.254.1    0.0.0.0         UG    1200   0        0 tun0
10.191.199.1    *               255.255.255.255 UH    14     0        0 eth0.4
10.191.199.49   *               255.255.255.255 UH    14     0        0 eth0.4
10.255.254.1    *               255.255.255.255 UH    0      0        0 tun0
10.255.254.2    *               255.255.255.255 UH    1200   0        0 tun0
34.120.255.244  *               255.255.255.255 UH    10     0        0 eth0.2
45.61.186.172   192.168.12.1    255.255.255.255 UGH   1      0        0 eth0.3
45.61.186.172   100.64.0.1      255.255.255.255 UGH   10     0        0 eth0.2
45.61.186.172   192.168.12.1    255.255.255.255 UGH   12     0        0 eth0.3
45.61.186.172   10.191.199.1    255.255.255.255 UGH   14     0        0 eth0.4
100.64.0.0      *               255.192.0.0     U     10     0        0 eth0.2
127.0.0.0       *               255.0.0.0       U     5      0        0 lo
192.168.12.0    *               255.255.255.0   U     12     0        0 eth0.3
192.168.100.1   dishy.starlink. 255.255.255.255 UGH   0      0        0 eth0.2
192.168.100.1   dishy.starlink. 255.255.255.255 UGH   10     0        0 eth0.2
  • what MPTCP parameters are you using? (sysctl net.mptcp)
net.mptcp.mptcp_binder_gateways =
net.mptcp.mptcp_checksum = 0
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = ecf
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0
  • if you are using fullmesh for the path manager, how did you configure it? what's the status? 
cat /proc/net/mptcp_fullmesh
Index, Address-ID, Backup, IP-address, if-idx
IPv4, next v4-index: 6
0, 1, 0, 192.168.12.226, 929
2, 3, 0, 10.191.199.49, 901
3, 4, 0, 100.96.159.149, 918
5, 6, 0, 10.255.254.2, 21
IPv6, next v6-index: 2
2, 10, 0, fd27:ca97:0230:0000:0000:0000:0000:0001, 901
3, 11, 0, fd27:ca97:0230:0100:0000:0000:0000:0001, 918
4, 12, 0, fd27:ca97:0230:0210:0000:0000:0000:0001, 929
  • what's your SELinux configuration? (try permissive or disable)

No SELinux involved

  • what's your firewall configuration?
config defaults
        option syn_flood '1'
        option forward 'REJECT'
        option input 'REJECT'
        option output 'REJECT'
        option disable_ipv6 '0'
        option drop_invalid '1'

config zone 'zone_lan'
        option name 'lan'
        option input 'ACCEPT'
        option output 'ACCEPT'
        option forward 'ACCEPT'
        option auto_helper '0'
        option mtu_fix '1'
        list network 'lan'

config zone 'zone_wan'
        option name 'wan'
        option input 'REJECT'
        option output 'ACCEPT'
        option forward 'REJECT'
        option masq '1'
        option mtu_fix '1'
        option auto_helper '0'
        list network 'Starlink'
        list network 'TMobile'
        list network 'ATT'

config forwarding
        option src 'lan'
        option dest 'wan'

config rule
        option name 'Allow-DHCP-Renew'
        option src 'wan'
        option proto 'udp'
        option dest_port '68'
        option target 'ACCEPT'
        option family 'ipv4'

config rule
        option name 'Allow-Ping'
        option src 'wan'
        option proto 'icmp'
        option icmp_type 'echo-request'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-IGMP'
        option src 'wan'
        option proto 'igmp'
        option family 'ipv4'
        option target 'ACCEPT'

config rule
        option name 'Allow-DHCPv6'
        option src 'wan'
        option proto 'udp'
        option src_ip 'fc00::/6'
        option dest_ip 'fc00::/6'
        option dest_port '546'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-MLD'
        option src 'wan'
        option proto 'icmp'
        option src_ip 'fe80::/10'
        list icmp_type '130/0'
        list icmp_type '131/0'
        list icmp_type '132/0'
        list icmp_type '143/0'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-ICMPv6-Forward'
        option src 'wan'
        option dest '*'
        option proto 'icmp'
        list icmp_type 'echo-request'
        list icmp_type 'echo-reply'
        list icmp_type 'destination-unreachable'
        list icmp_type 'packet-too-big'
        list icmp_type 'time-exceeded'
        list icmp_type 'bad-header'
        list icmp_type 'unknown-header-type'
        option limit '1000/sec'
        option family 'ipv6'
        option target 'ACCEPT'

config rule
        option name 'Allow-IPSec-ESP'
        option src 'wan'
        option dest 'lan'
        option proto 'esp'
        option target 'ACCEPT'

config rule
        option name 'Allow-ISAKMP'
        option src 'wan'
        option dest 'lan'
        option dest_port '500'
        option proto 'udp'
        option target 'ACCEPT'

config rule
        option name 'Support-UDP-Traceroute'
        option src 'wan'
        option dest_port '33434:33689'
        option proto 'udp'
        option family 'ipv4'
        option target 'REJECT'
        option enabled 'false'

config include
        option path '/etc/firewall.user'
        option reload '1'

config rule
        option enabled '1'
        option target 'ACCEPT'
        option name 'Allow-All-LAN-to-VPN'
        option dest 'vpn'
        option src 'lan'
        option proto 'all'

config rule
        option enabled '1'
        option target 'ACCEPT'
        option name 'Allow-All-Ping'
        option proto 'icmp'
        option dest '*'
        option src '*'
        option icmp_type 'echo-request'

config rule
        option enabled '1'
        option target 'ACCEPT'
        option name 'Allow-VPN-ICMP'
        option proto 'icmp'
        option src 'vpn'

config rule
        option enabled '1'
        option target 'ACCEPT'
        option name 'Allow-Lan-to-Wan'
        option dest 'wan'
        option src 'lan'
        option proto 'all'

config rule
        option enabled '1'
        option target 'ACCEPT'
        option name 'ICMPv6-Lan-to-OMR'
        option src 'lan'
        option family 'ipv6'
        option proto 'icmp'
        option limit '1000/sec'
        option icmp_type 'echo-reply destination-unreachable echo-request router-advertisement router-solicitation time-exceeded'

config include 'omr_server'
        option path '/etc/firewall.omr-server'
        option reload '1'

config include 'gre_tunnel'
        option path '/etc/firewall.gre-tunnel'
        option reload '0'

config include 'ttl'
        option path '/etc/firewall.ttl'
        option reload '1'

config forwarding 'fwlantovpn'
        option src 'lan'
        option dest 'vpn'

config rule 'blockquicproxy'
        option name 'Block QUIC Proxy'
        option proto 'udp'
        option dest_port '443'
        option target 'DROP'
        option src 'lan'

config rule 'blockquicall'
        option name 'Block QUIC All'
        option proto 'udp'
        option src '*'
        option dest '*'
        option dest_port '443'
        option target 'DROP'

config rule 'allowicmpipv6'
        option proto 'icmp'
        option target 'ACCEPT'
        option src 'wan'
        option name 'Allow IPv6 ICMP'
        option family 'ipv6'
        option icmp_type 'neighbour-advertisement neighbour-solicitation router-advertisement router-solicitation'

config rule 'allowdhcpv6546'
        option target 'ACCEPT'
        option src 'wan'
        option proto 'udp'
        option dest_port '547'
        option name 'Allow DHCPv6 (546-to-547)'
        option family 'ipv6'
        option src_port '546'

config rule 'allowdhcpv6547'
        option target 'ACCEPT'
        option src 'wan'
        option proto 'udp'
        option dest_port '546'
        option name 'Allow DHCPv6 (547-to-546)'
        option family 'ipv6'
        option src_port '547'

config include 'v2ray'
        option path '/etc/firewall.v2ray-rules'
        option reload '0'

config include 'omr_bypass'
        option path '/etc/firewall.omr-bypass'
        option reload '0'

config include 'ss_rules'
        option path '/etc/firewall.ss-rules'
        option reload '1'

config zone 'zone_vpn'
        option name 'vpn'
        option masq '1'
        option input 'REJECT'
        option forward 'ACCEPT'
        option output 'ACCEPT'
        option mtu_fix '1'
        option auto_helper '0'
        list network 'omrvpn'
        list network 'omr6in4'

config include 'miniupnpd'
        option type 'script'
        option path '/usr/share/miniupnpd/firewall.include'
        option family 'any'
        option reload '1'
matttbe commented 2 years ago

Hi @kevinh-csalabs

Thank you for the replies.

which IP routes/rules did you configure?

Do you have the IP routes and rules for IPv6 as well? Because I understood you have issues with IPv6.

ip -6 rule
ip -6 route table all

if you are using fullmesh for the path manager, how did you configure it? what's the status?

IPv6, next v6-index: 2
2, 10, 0, fd27:ca97:0230:0000:0000:0000:0000:0001, 901
3, 11, 0, fd27:ca97:0230:0100:0000:0000:0000:0001, 918
4, 12, 0, fd27:ca97:0230:0210:0000:0000:0000:0001, 929

Just to be sure, are these IPv6 "usable"? Maybe they are not and they should be marked as "no multipath"? Or maybe IPv6 on the corresponding interface should be disabled?

what's your firewall configuration?

I'm sorry, it is hard for me to read OpenWRT firewall configuration (I'm more used to ip6tables-save output). Do you know if the IPv6 is properly supported by OpenMPTCProuter?

After a very quick look in the kernel code, it seems more likely the issue comes from the routing rules that are not properly configured in IPv6. Our wiki explains what to do with IPv4 but it is similar with IPv6: https://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

kevinh-csalabs commented 2 years ago

Thank you for your help troubleshooting this – it’s much appreciated.

ip -6 rule

0: from all lookup local 1: from all fwmark 0x6539 lookup 6991337 1: from all fwmark 0x65395 lookup 65 1: from all fwmark 0x65391201 lookup 61201 1: from all fwmark 0x65391 lookup 61 32766: from all lookup main 4200000001: from all iif lo failed_policy 4200000002: from all iif eth0 failed_policy 4200000021: from all iif tun0 failed_policy 4200000831: from all iif 6in4-omr6in4 failed_policy 4200000901: from all iif eth0.4 failed_policy 4200001003: from all iif eth0.3 failed_policy 4200001011: from all iif eth0.2 failed_policy

It seems some policies are failing here but I’m unclear on why or how to diagnose…

Fullpath: Index, Address-ID, Backup, IP-address, if-idx IPv4, next v4-index: 7 0, 1, 0, 192.168.12.226, 1003 2, 3, 0, 10.191.199.49, 901 6, 7, 0, 100.96.159.149, 1011 IPv6, next v6-index: 3 2, 10, 0, fd27:ca97:0230:0100:0000:0000:0000:0001, 1011 A quick glance here and I’m only seeing one get set but there should be 3 connections…

ip6tables-save

Generated by ip6tables-save v1.8.7 on Thu Dec 1 22:20:18 2022

*raw :PREROUTING ACCEPT [5251057:3751662361] :OUTPUT ACCEPT [143140:13610095] COMMIT

Completed on Thu Dec 1 22:20:18 2022

Generated by ip6tables-save v1.8.7 on Thu Dec 1 22:20:18 2022

*nat :PREROUTING ACCEPT [34048:3855486] :INPUT ACCEPT [12794:1112252] :OUTPUT ACCEPT [68343:6889112] :POSTROUTING ACCEPT [88678:8452878] COMMIT

Completed on Thu Dec 1 22:20:18 2022

Generated by ip6tables-save v1.8.7 on Thu Dec 1 22:20:18 2022

*mangle :PREROUTING ACCEPT [138:12142] :INPUT ACCEPT [122:11054] :FORWARD ACCEPT [16:1088] :OUTPUT ACCEPT [120:16740] :POSTROUTING ACCEPT [136:17828] :omr-bypass6 - [0:0] :omr-bypass6-dpi - [0:0] -A PREROUTING -j omr-bypass6 -A INPUT -j omr-bypass6-dpi -A FORWARD -j omr-bypass6-dpi -A FORWARD -o eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone lan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i eth0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone lan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -o eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i eth0.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -o eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -o eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -o tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i tun0 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -o 6in4-omr6in4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A FORWARD -i 6in4-omr6in4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone vpn MTU fixing" -j TCPMSS --clamp-mss-to-pmtu -A omr-bypass6 -m set --match-set omr6_dst_bypass_eth0_4 dst -j MARK --set-xmark 0x653914/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_eth0_3 dst -j MARK --set-xmark 0x653912/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_eth0_2 dst -j MARK --set-xmark 0x653910/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_6in4-omr6in4 dst -j MARK --set-xmark 0x65391201/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_tun0 dst -j MARK --set-xmark 0x65391200/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_eth0 dst -j MARK --set-xmark 0x65396/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_lo dst -j MARK --set-xmark 0x65395/0xffffffff -A omr-bypass6 -m set --match-set omr6_dst_bypass_all dst -j MARK --set-xmark 0x6539/0xffffffff -A omr-bypass6-dpi -m ndpi --proto disneyplus -j MARK --set-xmark 0x6539/0xffffffff -A omr-bypass6-dpi -m mark --mark 0x6539 -j RETURN -A omr-bypass6-dpi -m ndpi --proto hulu -j MARK --set-xmark 0x6539/0xffffffff -A omr-bypass6-dpi -m mark --mark 0x6539 -j RETURN COMMIT

Completed on Thu Dec 1 22:20:18 2022

Generated by ip6tables-save v1.8.7 on Thu Dec 1 22:20:18 2022

*filter :INPUT DROP [0:0] :FORWARD DROP [0:0] :OUTPUT DROP [0:0] :MINIUPNPD - [0:0] :forwarding_lan_rule - [0:0] :forwarding_rule - [0:0] :forwarding_vpn_rule - [0:0] :forwarding_wan_rule - [0:0] :input_lan_rule - [0:0] :input_rule - [0:0] :input_vpn_rule - [0:0] :input_wan_rule - [0:0] :output_lan_rule - [0:0] :output_rule - [0:0] :output_vpn_rule - [0:0] :output_wan_rule - [0:0] :reject - [0:0] :syn_flood - [0:0] :zone_lan_dest_ACCEPT - [0:0] :zone_lan_forward - [0:0] :zone_lan_input - [0:0] :zone_lan_output - [0:0] :zone_lan_src_ACCEPT - [0:0] :zone_vpn_dest_ACCEPT - [0:0] :zone_vpn_forward - [0:0] :zone_vpn_input - [0:0] :zone_vpn_output - [0:0] :zone_vpn_src_REJECT - [0:0] :zone_wan_dest_ACCEPT - [0:0] :zone_wan_dest_REJECT - [0:0] :zone_wan_forward - [0:0] :zone_wan_input - [0:0] :zone_wan_output - [0:0] :zone_wan_src_REJECT - [0:0] -A INPUT -i lo -m comment --comment "!fw3" -j ACCEPT -A INPUT -m comment --comment "!fw3: Custom input rule chain" -j input_rule -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT -A INPUT -m conntrack --ctstate INVALID -m comment --comment "!fw3" -j DROP -A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "!fw3" -j syn_flood -A INPUT -i eth0 -m comment --comment "!fw3" -j zone_lan_input -A INPUT -i eth0.2 -m comment --comment "!fw3" -j zone_wan_input -A INPUT -i eth0.3 -m comment --comment "!fw3" -j zone_wan_input -A INPUT -i eth0.4 -m comment --comment "!fw3" -j zone_wan_input -A INPUT -i tun0 -m comment --comment "!fw3" -j zone_vpn_input -A INPUT -i 6in4-omr6in4 -m comment --comment "!fw3" -j zone_vpn_input -A INPUT -m comment --comment "!fw3" -j reject -A FORWARD -m comment --comment "!fw3: Custom forwarding rule chain" -j forwarding_rule -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT -A FORWARD -m conntrack --ctstate INVALID -m comment --comment "!fw3" -j DROP -A FORWARD -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m comment --comment "!fw3: Allow-All-Ping" -j ACCEPT -A FORWARD -p udp -m udp --dport 443 -m comment --comment "!fw3: Block QUIC All" -j DROP -A FORWARD -i eth0 -m comment --comment "!fw3" -j zone_lan_forward -A FORWARD -i eth0.2 -m comment --comment "!fw3" -j zone_wan_forward -A FORWARD -i eth0.3 -m comment --comment "!fw3" -j zone_wan_forward -A FORWARD -i eth0.4 -m comment --comment "!fw3" -j zone_wan_forward -A FORWARD -i tun0 -m comment --comment "!fw3" -j zone_vpn_forward -A FORWARD -i 6in4-omr6in4 -m comment --comment "!fw3" -j zone_vpn_forward -A FORWARD -m comment --comment "!fw3" -j reject -A OUTPUT -o lo -m comment --comment "!fw3" -j ACCEPT -A OUTPUT -m comment --comment "!fw3: Custom output rule chain" -j output_rule -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -m comment --comment "!fw3" -j ACCEPT -A OUTPUT -m conntrack --ctstate INVALID -m comment --comment "!fw3" -j DROP -A OUTPUT -o eth0 -m comment --comment "!fw3" -j zone_lan_output -A OUTPUT -o eth0.2 -m comment --comment "!fw3" -j zone_wan_output -A OUTPUT -o eth0.3 -m comment --comment "!fw3" -j zone_wan_output -A OUTPUT -o eth0.4 -m comment --comment "!fw3" -j zone_wan_output -A OUTPUT -o tun0 -m comment --comment "!fw3" -j zone_vpn_output -A OUTPUT -o 6in4-omr6in4 -m comment --comment "!fw3" -j zone_vpn_output -A OUTPUT -m comment --comment "!fw3" -j reject -A reject -p tcp -m comment --comment "!fw3" -j REJECT --reject-with tcp-reset -A reject -m comment --comment "!fw3" -j REJECT --reject-with icmp6-port-unreachable -A syn_flood -p tcp -m tcp --tcp-flags FIN,SYN,RST,ACK SYN -m limit --limit 25/sec --limit-burst 50 -m comment --comment "!fw3" -j RETURN -A syn_flood -m comment --comment "!fw3" -j DROP -A zone_lan_dest_ACCEPT -o eth0 -m comment --comment "!fw3" -j ACCEPT -A zone_lan_forward -m comment --comment "!fw3: Custom lan forwarding rule chain" -j forwarding_lan_rule -A zone_lan_forward -m comment --comment "!fw3: Allow-All-LAN-to-VPN" -j zone_vpn_dest_ACCEPT -A zone_lan_forward -m comment --comment "!fw3: Allow-Lan-to-Wan" -j zone_wan_dest_ACCEPT -A zone_lan_forward -m comment --comment "!fw3: Zone lan to wan forwarding policy" -j zone_wan_dest_ACCEPT -A zone_lan_forward -m comment --comment "!fw3: Zone lan to vpn forwarding policy" -j zone_vpn_dest_ACCEPT -A zone_lan_forward -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT -A zone_lan_input -m comment --comment "!fw3: Custom lan input rule chain" -j input_lan_rule -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 129 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m limit --limit 1000/sec -m comment --comment "!fw3: ICMPv6-Lan-to-OMR" -j ACCEPT -A zone_lan_input -p udp -m udp --dport 443 -m comment --comment "!fw3: Block QUIC Proxy" -j DROP -A zone_lan_input -m comment --comment "!fw3" -j zone_lan_src_ACCEPT -A zone_lan_output -m comment --comment "!fw3: Custom lan output rule chain" -j output_lan_rule -A zone_lan_output -m comment --comment "!fw3" -j zone_lan_dest_ACCEPT -A zone_lan_src_ACCEPT -i eth0 -m comment --comment "!fw3" -j ACCEPT -A zone_vpn_dest_ACCEPT -o tun0 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP -A zone_vpn_dest_ACCEPT -o tun0 -m comment --comment "!fw3" -j ACCEPT -A zone_vpn_dest_ACCEPT -o 6in4-omr6in4 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP -A zone_vpn_dest_ACCEPT -o 6in4-omr6in4 -m comment --comment "!fw3" -j ACCEPT -A zone_vpn_forward -m comment --comment "!fw3: Custom vpn forwarding rule chain" -j forwarding_vpn_rule -A zone_vpn_forward -j MINIUPNPD -A zone_vpn_forward -m comment --comment "!fw3" -j zone_vpn_dest_ACCEPT -A zone_vpn_input -m comment --comment "!fw3: Custom vpn input rule chain" -j input_vpn_rule -A zone_vpn_input -p ipv6-icmp -m comment --comment "!fw3: Allow-VPN-ICMP" -j ACCEPT -A zone_vpn_input -j MINIUPNPD -A zone_vpn_input -m comment --comment "!fw3" -j zone_vpn_src_REJECT -A zone_vpn_output -m comment --comment "!fw3: Custom vpn output rule chain" -j output_vpn_rule -A zone_vpn_output -m comment --comment "!fw3" -j zone_vpn_dest_ACCEPT -A zone_vpn_src_REJECT -i tun0 -m comment --comment "!fw3" -j reject -A zone_vpn_src_REJECT -i 6in4-omr6in4 -m comment --comment "!fw3" -j reject -A zone_wan_dest_ACCEPT -o eth0.2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP -A zone_wan_dest_ACCEPT -o eth0.2 -m comment --comment "!fw3" -j ACCEPT -A zone_wan_dest_ACCEPT -o eth0.3 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP -A zone_wan_dest_ACCEPT -o eth0.3 -m comment --comment "!fw3" -j ACCEPT -A zone_wan_dest_ACCEPT -o eth0.4 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP -A zone_wan_dest_ACCEPT -o eth0.4 -m comment --comment "!fw3" -j ACCEPT -A zone_wan_dest_REJECT -o eth0.2 -m comment --comment "!fw3" -j reject -A zone_wan_dest_REJECT -o eth0.3 -m comment --comment "!fw3" -j reject -A zone_wan_dest_REJECT -o eth0.4 -m comment --comment "!fw3" -j reject -A zone_wan_forward -m comment --comment "!fw3: Custom wan forwarding rule chain" -j forwarding_wan_rule -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 128 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 129 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 1 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 2 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 3 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 4/0 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p ipv6-icmp -m icmp6 --icmpv6-type 4/1 -m limit --limit 1000/sec -m comment --comment "!fw3: Allow-ICMPv6-Forward" -j ACCEPT -A zone_wan_forward -p esp -m comment --comment "!fw3: Allow-IPSec-ESP" -j zone_lan_dest_ACCEPT -A zone_wan_forward -p udp -m udp --dport 500 -m comment --comment "!fw3: Allow-ISAKMP" -j zone_lan_dest_ACCEPT -A zone_wan_forward -m comment --comment "!fw3" -j zone_wan_dest_REJECT -A zone_wan_input -m comment --comment "!fw3: Custom wan input rule chain" -j input_wan_rule -A zone_wan_input -s fc00::/6 -d fc00::/6 -p udp -m udp --dport 546 -m comment --comment "!fw3: Allow-DHCPv6" -j ACCEPT -A zone_wan_input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 130/0 -m comment --comment "!fw3: Allow-MLD" -j ACCEPT -A zone_wan_input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 131/0 -m comment --comment "!fw3: Allow-MLD" -j ACCEPT -A zone_wan_input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 132/0 -m comment --comment "!fw3: Allow-MLD" -j ACCEPT -A zone_wan_input -s fe80::/10 -p ipv6-icmp -m icmp6 --icmpv6-type 143/0 -m comment --comment "!fw3: Allow-MLD" -j ACCEPT -A zone_wan_input -p ipv6-icmp -m icmp6 --icmpv6-type 136 -m comment --comment "!fw3: Allow IPv6 ICMP" -j ACCEPT -A zone_wan_input -p ipv6-icmp -m icmp6 --icmpv6-type 135 -m comment --comment "!fw3: Allow IPv6 ICMP" -j ACCEPT -A zone_wan_input -p ipv6-icmp -m icmp6 --icmpv6-type 134 -m comment --comment "!fw3: Allow IPv6 ICMP" -j ACCEPT -A zone_wan_input -p ipv6-icmp -m icmp6 --icmpv6-type 133 -m comment --comment "!fw3: Allow IPv6 ICMP" -j ACCEPT -A zone_wan_input -p udp -m udp --sport 546 --dport 547 -m comment --comment "!fw3: Allow DHCPv6 (546-to-547)" -j ACCEPT -A zone_wan_input -p udp -m udp --sport 547 --dport 546 -m comment --comment "!fw3: Allow DHCPv6 (547-to-546)" -j ACCEPT -A zone_wan_input -m comment --comment "!fw3" -j zone_wan_src_REJECT -A zone_wan_output -m comment --comment "!fw3: Custom wan output rule chain" -j output_wan_rule -A zone_wan_output -m comment --comment "!fw3" -j zone_wan_dest_ACCEPT -A zone_wan_src_REJECT -i eth0.2 -m comment --comment "!fw3" -j reject -A zone_wan_src_REJECT -i eth0.3 -m comment --comment "!fw3" -j reject -A zone_wan_src_REJECT -i eth0.4 -m comment --comment "!fw3" -j reject COMMIT

Completed on Thu Dec 1 22:20:18 2022

In regards to the routing rules theory, that may be correct. I’m not super familiar with the underpinnings of the openmptcprouter distro, but I’m sure it is setting the routes automatically. Assuming they are setting the routes automatically this could be the problem – glancing at the automatic script solution it looks like only ipv4 is being handled? https://github.com/multipath-tcp/mptcp-scripts/blob/master/scripts/rt_table/mptcp_up

matttbe commented 1 year ago 
ip -6 rule

0:      from all lookup local
1:      from all fwmark 0x6539 lookup 6991337
1:      from all fwmark 0x65395 lookup 65
1:      from all fwmark 0x65391201 lookup 61201
1:      from all fwmark 0x65391 lookup 61
32766:  from all lookup main
4200000001:     from all iif lo failed_policy
4200000002:     from all iif eth0 failed_policy
4200000021:     from all iif tun0 failed_policy
4200000831:     from all iif 6in4-omr6in4 failed_policy
4200000901:     from all iif eth0.4 failed_policy
4200001003:     from all iif eth0.3 failed_policy
4200001011:     from all iif eth0.2 failed_policy

It seems some policies are failing here but I’m unclear on why or how to diagnose…

I think it is normal to see these failed_policy on OpenWRT.

Do you mind also share the second part please? (ip -6 route table all) With only the 'rules', it looks like the routing rules for IPv6 are missing. I don't see any from <IP> lookup <table>. You can compare with IPv4 (using -4 instead of -6);

ip6tables-save

The IPTables rules seem to allow outgoing traffic for many Ethernet interfaces (all? eth0{,.[234]}) except if the ConnTrack status is not OK but I guess it is fine except if you have a transparent proxy but I guess no.

In regards to the routing rules theory, that may be correct. I’m not super familiar with the underpinnings of the openmptcprouter distro, but I’m sure it is setting the routes automatically. Assuming they are setting the routes automatically this could be the problem – glancing at the automatic script solution it looks like only ipv4 is being handled? https://github.com/multipath-tcp/mptcp-scripts/blob/master/scripts/rt_table/mptcp_up

Did you check if OpenMPTCProuter fully supports IPv6? cc: @Ysurac

Setting the routes manually is not difficult: you just need to ensure that the traffic from a certain IP to going to use the interface linked to this IP, e.g.

ip -6 rule add from <IP> table <number>
ip -6 route add default via <NEXT HOP> [dev <iface>] table <number>

For each interface (the one linked to the default route is optional if the default route doesn't change)

For more details: https://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

kevinh-csalabs commented 1 year ago

ip route show table all yields: default via 100.64.0.1 dev eth0.2 table 10 100.64.0.0/10 dev eth0.2 table 10 scope link default via 192.168.12.1 dev eth0.3 table 12 192.168.12.0/24 dev eth0.3 table 12 scope link default via 10.191.199.1 dev eth0.4 table 14 10.191.199.49 dev eth0.4 table 14 scope link 192.168.42.0/24 dev eth0 table lan proto static scope link metric 6 local default dev lo table 100 scope host default via 192.168.12.1 dev eth0.3 table 991337 10.255.254.2 dev tun0 table 1200 scope link default via 10.255.254.1 dev tun0 default via 192.168.12.1 dev eth0.3 metric 1 default via 100.64.0.1 dev eth0.2 metric 10 default via 192.168.12.1 dev eth0.3 metric 12 default via 10.191.199.1 dev eth0.4 metric 14 10.191.199.1 dev eth0.4 proto static scope link src 10.191.199.49 metric 14 10.191.199.49 dev eth0.4 scope link metric 14 10.255.254.1 dev tun0 proto kernel scope link src 10.255.254.2 10.255.254.2 dev tun0 scope link metric 1200 34.120.255.244 dev eth0.2 proto static scope link src 100.96.159.149 metric 10 45.61.186.172 via 192.168.12.1 dev eth0.3 metric 1 45.61.186.172 via 100.64.0.1 dev eth0.2 metric 10 45.61.186.172 via 192.168.12.1 dev eth0.3 metric 12 45.61.186.172 via 10.191.199.1 dev eth0.4 metric 14 100.64.0.0/10 dev eth0.2 scope link metric 10 127.0.0.0/8 dev lo proto static scope link metric 5 192.168.12.0/24 dev eth0.3 scope link metric 12 192.168.100.1 via 192.168.100.1 dev eth0.2 192.168.100.1 via 192.168.100.1 dev eth0.2 proto static metric 10 local 10.191.199.49 dev eth0.4 table local proto kernel scope host src 10.191.199.49 local 10.255.254.2 dev tun0 table local proto kernel scope host src 10.255.254.2 broadcast 100.64.0.0 dev eth0.2 table local proto kernel scope link src 100.96.159.149 local 100.96.159.149 dev eth0.2 table local proto kernel scope host src 100.96.159.149 broadcast 100.127.255.255 dev eth0.2 table local proto kernel scope link src 100.96.159.149 broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1 broadcast 192.168.12.0 dev eth0.3 table local proto kernel scope link src 192.168.12.226 local 192.168.12.226 dev eth0.3 table local proto kernel scope host src 192.168.12.226 broadcast 192.168.12.255 dev eth0.3 table local proto kernel scope link src 192.168.12.226 broadcast 192.168.42.0 dev eth0 table local proto kernel scope link src 192.168.42.1 local 192.168.42.1 dev eth0 table local proto kernel scope host src 192.168.42.1 broadcast 192.168.42.255 dev eth0 table local proto kernel scope link src 192.168.42.1 default from fe80::a00:0/126 dev 6in4-omr6in4 proto static metric 1201 pref medium fd27:ca97:230:200:417:b6ff:fe09:abc3 dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200:d13:7949:9802:ccf2 dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200:141d:ab8:b338:293f dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200:18f3:9fc:133c:22a0 dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200:6057:f1ff:fefc:5f51 dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200:c82c:935c:89a6:8369 dev eth0 proto static metric 1024 pref medium fd27:ca97:230:200::/64 dev eth0 proto static metric 6 pref medium fd27:ca97:230:210::/64 dev eth0.3 proto static metric 12 pref medium fd27:ca97:230:211::/64 dev tun0 proto static metric 1200 pref medium unreachable fd27:ca97:230::/48 dev lo proto static metric 2147483647 pref medium fe80::a00:1 dev 6in4-omr6in4 metric 1024 pref medium fe80::a00:0/126 dev 6in4-omr6in4 proto static metric 1201 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev 6in4-omr6in4 proto kernel metric 256 pref medium fe80::/64 dev eth0.3 proto kernel metric 256 pref medium fe80::/64 dev tun0 proto kernel metric 256 pref medium default via fe80::a00:1 dev 6in4-omr6in4 metric 1 pref medium local ::1 dev lo table local proto kernel metric 0 pref medium anycast fd27:ca97:230:200:: dev eth0 table local proto kernel metric 0 pref medium local fd27:ca97:230:200::1 dev eth0 table local proto kernel metric 0 pref medium anycast fd27:ca97:230:210:: dev eth0.3 table local proto kernel metric 0 pref medium local fd27:ca97:230:210::1 dev eth0.3 table local proto kernel metric 0 pref medium anycast fd27:ca97:230:211:: dev tun0 table local proto kernel metric 0 pref medium local fd27:ca97:230:211::1 dev tun0 table local proto kernel metric 0 pref medium anycast fe80:: dev 6in4-omr6in4 table local proto kernel metric 0 pref medium anycast fe80:: dev eth0 table local proto kernel metric 0 pref medium anycast fe80:: dev eth0.3 table local proto kernel metric 0 pref medium anycast fe80:: dev tun0 table local proto kernel metric 0 pref medium anycast fe80::a00:0 dev 6in4-omr6in4 table local proto kernel metric 0 pref medium local fe80::a00:2 dev 6in4-omr6in4 table local proto kernel metric 0 pref medium local fe80::aff:fe02 dev 6in4-omr6in4 table local proto kernel metric 0 pref medium local fe80::b5fd:bae8:7f3f:e769 dev tun0 table local proto kernel metric 0 pref medium local fe80::e65f:1ff:fea3:d95 dev eth0 table local proto kernel metric 0 pref medium local fe80::e65f:1ff:fea3:d95 dev eth0.3 table local proto kernel metric 0 pref medium multicast ff00::/8 dev eth0 table local proto kernel metric 256 pref medium multicast ff00::/8 dev 6in4-omr6in4 table local proto kernel metric 256 pref medium multicast ff00::/8 dev eth0.3 table local proto kernel metric 256 pref medium multicast ff00::/8 dev tun0 table local proto kernel metric 256 pref medium

From what I’ve read openmptcprouter should support ipv6 fully but I have been running into some issues (this being one of them)

Regarding outbound traffic – it is designed to send all traffic through a tunnel to the vps unless the vps is unavailable in which case it load balances outbound traffic. LAN and WAN share a physical port because this is on a raspberry pi 4 connected to a switch, but WANs are on different VLANs to separate the traffic.

matttbe commented 1 year ago

ip route show table all yields: default via 100.64.0.1 dev eth0.2 table 10 100.64.0.0/10 dev eth0.2 table 10 scope link default via 192.168.12.1 dev eth0.3 table 12 192.168.12.0/24 dev eth0.3 table 12 scope link default via 10.191.199.1 dev eth0.4 table 14 10.191.199.49 dev eth0.4 table 14 scope link

We cannot see these kinds of rules in IPv6. Then the kernel tries to send the SYN for the new subflows in IPv6 it just created to the wrong interface and fails.

From what I’ve read openmptcprouter should support ipv6 fully but I have been running into some issues (this being one of them)

Maybe something to enable or configure? Sorry I don't know how it works. Because it doesn't look like there is an issue on the kernel side, it is probably best to open a new ticket on OpenMPTCPRouter side and close this one if that's alright.

Regarding outbound traffic – it is designed to send all traffic through a tunnel to the vps unless the vps is unavailable in which case it load balances outbound traffic. LAN and WAN share a physical port because this is on a raspberry pi 4 connected to a switch, but WANs are on different VLANs to separate the traffic.

OK but I guess the issue you have is due to the routing rules not being configured