mwan3 marks second IPv6 link as down

[dersch81]

OpenWRT 18.06.1 r7258-5eb055306f mwan3: 2.6.18-1 luci-app-mwan3: git-18.340.83383-3dea6b5-1

Maintainer: @feckert (find it by checking history of the package Makefile)

Description: It looks like the mwan3 package is not able to handle 2 ipv6 uplinks at the same time. On interface is always marked as down. That is also resulting in totally no IPv6 route for the clients of the router. The router itself keeps his IPv6 connectivity. After changing some settings it is working for some seconds, but right after mwan3 marks the second IPv6 interface as down it breaks the IPv6 routes.

LOG:

Sun Dec  9 22:57:25 2018 user.notice mwan3[4045]: Starting tracker on interface wan2_6 (eth1.111)
Sun Dec  9 22:57:25 2018 daemon.err uhttpd[1161]: luci: accepted login on / for root from 192.168.10.120
Sun Dec  9 22:57:26 2018 daemon.info odhcpd[1043]: Using a RA lifetime of 1800 seconds on br-lan
Sun Dec  9 22:57:26 2018 user.info mwan3track[4283]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan2_6 (eth1.111)
Sun Dec  9 22:57:27 2018 kern.notice kernel: [   66.802506] random: crng init done
Sun Dec  9 22:57:28 2018 user.info mwan3track[2339]: Lost 2 ping(s) on interface wan2 (eth1.111)
Sun Dec  9 22:57:32 2018 user.info mwan3track[4283]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan2_6 (eth1.111)
Sun Dec  9 22:57:32 2018 user.info mwan3[4045]: connection tracking not flushed on interface wan2_6 (eth1.111) ifup
Sun Dec  9 22:57:33 2018 user.notice firewall: Reloading firewall due to ifup of wan2_6 (eth1.111)
Sun Dec  9 22:57:36 2018 user.notice mwan3[4839]: Execute ifup event on interface wan_6 (eth0.13)
Sun Dec  9 22:57:37 2018 user.info mwan3track[4283]: Check (ping) failed for target "2001:4860:4860::8888" on interface wan2_6 (eth1.111)
Sun Dec  9 22:57:37 2018 user.notice mwan3track[4283]: Interface wan2_6 (eth1.111) is offline
Sun Dec  9 22:57:38 2018 user.notice mwan3[4839]: Starting tracker on interface wan_6 (eth0.13)
Sun Dec  9 22:57:39 2018 daemon.info odhcpd[1043]: Using a RA lifetime of 1800 seconds on br-lan
Sun Dec  9 22:57:45 2018 user.info mwan3[4839]: connection tracking not flushed on interface wan_6 (eth0.13) ifup
Sun Dec  9 22:57:47 2018 user.notice firewall: Reloading firewall due to ifup of wan_6 (eth0.13)
Sun Dec  9 22:57:47 2018 user.notice mwan3[5088]: Execute ifdown event on interface wan2_6 (eth1.111)
Sun Dec  9 22:57:47 2018 user.info mwan3track[2969]: Check (ping) failed for target "1.1.1.1" on interface wan3 (eth1.222)
Sun Dec  9 22:57:52 2018 user.info mwan3track[2969]: Lost 1 ping(s) on interface wan3 (eth1.222)
Sun Dec  9 22:57:54 2018 user.info mwan3[5088]: connection tracking not flushed on interface wan2_6 (eth1.111) ifdown
Sun Dec  9 22:57:54 2018 user.info mwan3track[4283]: Detect ifdown event on interface wan2_6 (eth1.111)

mwan3 details:

Interface status:
 interface wan2 is online and tracking is active
 interface wan is online and tracking is active
 interface wan3 is online and tracking is active
 interface wan2_6 is offline and tracking is active
 interface wan_6 is online and tracking is active

Current ipv4 policies:
balanced:
 wan3 (33%)
 wan (33%)
 wan2 (33%)

wan2_wan:
 wan2 (100%)

wan_wan2:
 wan3 (50%)
 wan (50%)

Current ipv6 policies:
balanced:
 wan_6 (100%)

wan2_wan:
 wan_6 (100%)

wan_wan2:
 wan_6 (100%)

Directly connected ipv4 networks:
 192.168.13.4
 10.10.11.255
 10.10.11.2
 10.10.10.0/24
 127.0.0.1
 192.168.13.0
 224.0.0.0/3
 127.0.0.0
 192.168.33.0
 10.10.11.0/24
 192.168.10.1
 192.168.13.0/24
 192.168.33.0/24
 127.0.0.0/8
 10.10.10.255
 10.10.10.254
 192.168.13.255
 10.10.11.0
 192.168.33.2
 10.10.10.0
 127.255.255.255
 192.168.33.255

Directly connected ipv6 networks:
 2001:470:74d7:fff7::/64
 fe80::/64
 fd3e:42d3:5404::/64
 2003:e6:c3d2:83fc::/64
 2003:e6:c3d2:8300::/64
 2003:e6:c3d2:8300::/56
 2001:470:74d7:13::/64

Active ipv4 user rules:
   12   736 S T_UM_Failover  all  --  *      *       0.0.0.0/0            0.0.0.0/0            
    0     0 S Balanced  all  --  *      *       0.0.0.0/0            0.0.0.0/0            
    0     0 S UM_T_Failover  all  --  *      *       0.0.0.0/0            0.0.0.0/0            

Active ipv6 user rules:
  334 37565 S T_UM_Failover  all      *      *       ::/0                 ::/0                 
    0     0 S Balanced  all      *      *       ::/0                 ::/0                 
    0     0 S UM_T_Failover  all      *      *       ::/0                 ::/0

Trace of a client:

traceroute to openwrt.org (2a03:b0c0:3:d0::1af1:1), 30 hops max, 64 byte packets
 1  2003:e6:c3d2:8380::1  0.968 ms !N

Trace of the main router

traceroute to lede-project.org (2a03:b0c0:3:d0::1af1:1), 30 hops max, 64 byte packets
 1  2003:e6:c3d2:8300:9ec7:a6ff:fe32:7045  0.884 ms
 2  2003:e6:c3bf:1282::1  5.610 ms
 3  2003:0:1307:400a::1  5.800 ms
 4  2003:0:1307:400a::2  5.453 ms
 5  2001:418:0:5000::326  5.499 ms
 6  *
 7  2a03:b0c0:3:d0::1af1:1  6.290 ms

[dersch81]

I notice the same behaivor with another router where i only have one IPv6 uplink. It is marked as down. The weird thing is that the Router descriped beyond with two IPv6 uplinks only one is marked as down.

[luizluca]

I'm not mwan3 maintainer but I'll have the same setup next year. Could you share your network config? Could you test ping6 -I eth1.111 2001:4860:4860::8888 with and without mwan3 started? A packet sniffer is your friend.

[feckert]

as mentioned by @luizluca could you send use your network config?

[BIGFAAT]

I can see the same problem at home. I cant use 2 ipv6 wan connections at the same time. One will be always marked offline. Tell me if you want to see some configs.

[feckert]

@BIGFAAT network/mwan3 config would be nice

[jonjonw]

I'm having similar issues so I'll chime in. I have a regular ipv4 connection and a 6in4 tunnel. The ipv4 connection has no issues but the 6in4 tunnel works for a while until mwan3 marks it as down (usually 30 seconds up then 30 seconds down and repeat).

My /etc/config/network:

config interface 'loopback'
    option ifname 'lo'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'

config globals 'globals'
    option ula_prefix 'fd0b:x:x::/48'

config interface 'lan'
    option type 'bridge'
    option ifname 'eth1.1'
    option proto 'static'
    option ipaddr '192.168.1.1'
    option netmask '255.255.255.0'
    option ip6assign '60'

config interface 'wan'
    option ifname 'eth0.2'
    option proto 'dhcp'

config interface 'wan6'
    option ifname 'eth0.2'
    option proto '6in4'
    option tunlink 'wan'
    option username 'mfire'
    option peeraddr '216.x.x.x'
    list ip6prefix '2001:x:x:x::/64'
    option ip6addr '2001:x:x:x::x/64'
    option tunnelid 'XXX'
    option password 'XXX'

config switch
    option name 'switch0'
    option reset '1'
    option enable_vlan '1'

config switch_vlan
    option device 'switch0'
    option vlan '1'
    option ports '2 3 4 5 0t'

config switch_vlan
    option device 'switch0'
    option vlan '2'
    option ports '1 6t'

# ping6 -I 6in4-wan6 2001:4860:4860::8888
PING 2001:4860:4860::8888 (2001:4860:4860::8888): 56 data bytes
^C
--- 2001:4860:4860::8888 ping statistics ---
9 packets transmitted, 0 packets received, 100% packet loss

I could be wrong but the routes seem a bit strange to me (I suspect the last one gets added by he.net?):

# route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         192.x.x.65  0.0.0.0         UG    0      0        0 eth0.2
192.x.x.64  *               255.255.255.192 U     0      0        0 eth0.2
192.168.1.0     *               255.255.255.0   U     0      0        0 br-lan
216.x.x.x 192.x.x.65  255.255.255.255 UGH   0      0        0 eth0.2

So I tried deleting the offending route and now the ping works:

# route del -net 216.x.x.x netmask 255.255.255.255
# ping6 -I 6in4-wan6 2001:4860:4860::8888
PING 2001:4860:4860::8888 (2001:4860:4860::8888): 56 data bytes
64 bytes from 2001:4860:4860::8888: seq=0 ttl=58 time=37.942 ms
64 bytes from 2001:4860:4860::8888: seq=1 ttl=58 time=40.788 ms
64 bytes from 2001:4860:4860::8888: seq=2 ttl=58 time=34.520 ms
64 bytes from 2001:4860:4860::8888: seq=3 ttl=58 time=35.228 ms
64 bytes from 2001:4860:4860::8888: seq=4 ttl=58 time=35.772 ms
^C
--- 2001:4860:4860::8888 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 34.520/36.850/40.788 ms

However, mwan3 still alternates between marking the interface as up and down every 30 seconds.

My /etc/config/mwan3 is pretty close to the default (other than enabling wan6:

config globals 'globals'
    option mmx_mask '0x3F00'
    option rtmon_interval '5'

config interface 'wan'
    option enabled '1'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option family 'ipv4'
    option reliability '2'
    option count '1'
    option timeout '2'
    option failure_latency '1000'
    option recovery_latency '500'
    option failure_loss '20'
    option recovery_loss '5'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wan6'
    list track_ip '2001:4860:4860::8844'
    list track_ip '2001:4860:4860::8888'
    list track_ip '2620:0:ccd::2'
    list track_ip '2620:0:ccc::2'
    option family 'ipv6'
    option reliability '2'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'
    option enabled '1'
    option initial_state 'online'
    option track_method 'ping'
    option size '56'
    option max_ttl '60'
    option check_quality '0'
    option failure_interval '5'
    option recovery_interval '5'

config interface 'wanb'
    option enabled '0'
    list track_ip '8.8.4.4'
    list track_ip '8.8.8.8'
    list track_ip '208.67.222.222'
    list track_ip '208.67.220.220'
    option family 'ipv4'
    option reliability '1'
    option count '1'
    option timeout '2'
    option failure_latency '1000'
    option recovery_latency '500'
    option failure_loss '20'
    option recovery_loss '5'
    option interval '5'
    option down '3'
    option up '8'

config interface 'wanb6'
    option enabled '0'
    list track_ip '2001:4860:4860::8844'
    list track_ip '2001:4860:4860::8888'
    list track_ip '2620:0:ccd::2'
    list track_ip '2620:0:ccc::2'
    option family 'ipv6'
    option reliability '1'
    option count '1'
    option timeout '2'
    option interval '5'
    option down '3'
    option up '8'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wanb_m1_w2'
    option interface 'wanb'
    option metric '1'
    option weight '2'

config member 'wanb_m2_w2'
    option interface 'wanb'
    option metric '2'
    option weight '2'

config member 'wan6_m1_w3'
    option interface 'wan6'
    option metric '1'
    option weight '3'

config member 'wan6_m2_w3'
    option interface 'wan6'
    option metric '2'
    option weight '3'

config member 'wanb6_m1_w2'
    option interface 'wanb6'
    option metric '1'
    option weight '2'

config member 'wanb6_m2_w2'
    option interface 'wanb6'
    option metric '2'
    option weight '2'

config policy 'wan_only'
    list use_member 'wan_m1_w3'
    list use_member 'wan6_m1_w3'

config policy 'wanb_only'
    list use_member 'wanb_m1_w2'
    list use_member 'wanb6_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wanb_m1_w2'
    list use_member 'wan6_m1_w3'
    list use_member 'wanb6_m1_w2'

config policy 'wan_wanb'
    list use_member 'wan_m1_w3'
    list use_member 'wanb_m2_w2'
    list use_member 'wan6_m1_w3'
    list use_member 'wanb6_m2_w2'

config policy 'wanb_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wanb_m1_w2'
    list use_member 'wan6_m2_w3'
    list use_member 'wanb6_m1_w2'

config rule 'https'
    option sticky '1'
    option dest_port '443'
    option proto 'tcp'
    option use_policy 'balanced'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option use_policy 'balanced'

[jamesmacwhite]

@jonjonw Your 6in4 issues should be resolved in mwan3 2.8.4.

@dersch81 I think you managed to sort your issue in another discussion where we explored the issues with IPv6 interfaces having a /64 and /128 address, which @brianjmurrell resolved in his PR that got merged.

@BIGFAAT Without network configs, no one can really help, but try mwan3 2.8.4 IPv6 interfaces should be better supported.

@feckert Could close this one?

[feckert]

@jamesmacwhite thanks for this