mwan3: 2.10.x branch 50% ping loss with L2TP IPv6

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md

GNU General Public License v2.0

3.94k stars 3.46k forks source link

mwan3: 2.10.x branch 50% ping loss with L2TP IPv6 #14332

Open jamesmacwhite opened 3 years ago

jamesmacwhite commented 3 years ago

Maintainer: @aaronjg @feckert Environment: OpenWrt 19.07.5 mwan3 2.10.5

Description:

Under the 2.10.x branch my IPv6 setup seems to be experiencing issues. I've dug into it a little bit and found at least one symptom I can see is pings to IPv6 addresses i.e. ipv6.google.com from LAN clients are timing out 50% of the time. All interfaces are reporting an up status within mwan3, so mwan3track doesn't seem to think there's any issue with the interfaces.

This is a sample continuous IPv6 ping from a Windows client:

Pinging ipv6.l.google.com [2404:6800:4005:808::200e] with 32 bytes of data:
Reply from 2404:6800:4005:808::200e: time=282ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=277ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=278ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=279ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=282ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=279ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=282ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=283ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=282ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=287ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=278ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=284ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=281ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=313ms
Request timed out.
Reply from 2404:6800:4005:808::200e: time=299ms
Request timed out.

Ping statistics for 2404:6800:4005:808::200e:
    Packets: Sent = 30, Received = 15, Lost = 15 (50% loss),
Approximate round trip times in milli-seconds:
    Minimum = 277ms, Maximum = 313ms, Average = 284ms

I've been able to replicate this on another Windows client with the same behaviour.

Doing the same continuous IPv6 ping from the router itself with mwan3 enabled, no ping loss seems to occur.

root@linksys-wrt3200acm:~# ping ipv6.google.com
PING ipv6.google.com (2404:6800:4005:808::200e): 56 data bytes
64 bytes from 2404:6800:4005:808::200e: seq=0 ttl=113 time=279.628 ms
64 bytes from 2404:6800:4005:808::200e: seq=1 ttl=113 time=282.981 ms
64 bytes from 2404:6800:4005:808::200e: seq=2 ttl=113 time=274.847 ms
64 bytes from 2404:6800:4005:808::200e: seq=3 ttl=113 time=278.284 ms
64 bytes from 2404:6800:4005:808::200e: seq=4 ttl=113 time=275.490 ms
64 bytes from 2404:6800:4005:808::200e: seq=5 ttl=113 time=290.525 ms
64 bytes from 2404:6800:4005:808::200e: seq=6 ttl=113 time=275.543 ms
64 bytes from 2404:6800:4005:808::200e: seq=7 ttl=113 time=275.152 ms
64 bytes from 2404:6800:4005:808::200e: seq=8 ttl=113 time=279.917 ms
64 bytes from 2404:6800:4005:808::200e: seq=9 ttl=113 time=275.959 ms
64 bytes from 2404:6800:4005:808::200e: seq=10 ttl=113 time=277.035 ms
64 bytes from 2404:6800:4005:808::200e: seq=11 ttl=113 time=276.082 ms
64 bytes from 2404:6800:4005:808::200e: seq=12 ttl=113 time=276.150 ms
64 bytes from 2404:6800:4005:808::200e: seq=13 ttl=113 time=275.293 ms
64 bytes from 2404:6800:4005:808::200e: seq=14 ttl=113 time=301.525 ms
64 bytes from 2404:6800:4005:808::200e: seq=15 ttl=113 time=302.579 ms
64 bytes from 2404:6800:4005:808::200e: seq=16 ttl=113 time=274.776 ms
64 bytes from 2404:6800:4005:808::200e: seq=17 ttl=113 time=284.501 ms
64 bytes from 2404:6800:4005:808::200e: seq=18 ttl=113 time=303.208 ms
64 bytes from 2404:6800:4005:808::200e: seq=19 ttl=113 time=284.229 ms
64 bytes from 2404:6800:4005:808::200e: seq=20 ttl=113 time=287.108 ms
64 bytes from 2404:6800:4005:808::200e: seq=21 ttl=113 time=277.517 ms
64 bytes from 2404:6800:4005:808::200e: seq=22 ttl=113 time=285.347 ms
64 bytes from 2404:6800:4005:808::200e: seq=23 ttl=113 time=276.671 ms
64 bytes from 2404:6800:4005:808::200e: seq=24 ttl=113 time=277.192 ms
^C
--- ipv6.google.com ping statistics ---
25 packets transmitted, 25 packets received, 0% packet loss
round-trip min/avg/max = 274.776/281.901/303.208 ms

The track_ips for the ping test for the primary IPv6 interface seems to be OK as well:

However it appears LAN clients seem to have a 50% loss on IPv6 traffic, so this is causing long load times and sometimes websites not loading when using IPv6. IPv4 seems OK.

This doesn't happen with the same setup on the 2.8.x branch, so one or more changes between the branches seems to be having a negative impact with my IPv6 configuration. Which is why I have not moved to running the newer 2.10.x branch mwan3 package yet.

Let me know what further information I can provide to try and debug the issue. I don't feel this explains the root cause but a symptom of the problem so to speak. It is at least a little clearer than websites just timing out, so it's one step closer.

aaronjg commented 3 years ago

Thanks for the diagnostics. The pings to ipv6.google.com seem really slow, especially compared to the tracking IPs. Can you try pinging the tracking IPs the same way you are pinging ipv6.google.com and see if they have the same latency?

Have you set this up as failover or load balancing? If load balancing, can you change to failover and see if the issue persists. If failover, can you try changing the policy so there is only one ipv6 member?

Also, with the 2.8.x version, can you use tcpdump on the router to make sure that it is actually working as expected - that all ping echo ICMP packets are leaving the expected IPV6 interface and none are going out any other interface?

Is the latency as bad with the 2.8.x version as 2.10?

jamesmacwhite commented 3 years ago

The latency did seem high but it was run right after an upgrade of the mwan3 package, so the system load might have been higher than average, I think after a few minutes it did start to normalise.

Most of my configuration is failover with 3 physical WAN connections. I do use load balancing on a couple of Wireguard interfaces but they aren't used by default. The primary IPv6 interface was being used to test with the IPv6 pings and is configured with failover WAN -> WANB -> WANC

Latency isn't as bad on 2.8.x, but the latency being high might have been system load related. However, the 50% loss seemed to be consistent, so that was the main thing to flag.

jamesmacwhite commented 3 years ago

Using tcpdump just targeting ICMP6 traffic. This was the result of the IPv6 ping test to ipv6.google.com which lhr48s20-in-x0e.1e100.net is a CNAME alias of. I've confirmed it's the same for any IPv6 address, so tracking IP etc. The 50% loss is consistent on clients, it doesn't appear to happen for pings from the router directly though, so maybe that's a starting point.

tcpdump when pinging ipv6.google.com from a LAN client.

# tcpdump -p icmp6 (filtered to lhr48s20-in-x0e.1e100.net)
10:06:51.604401 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 261, length 40
10:06:51.622531 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 261, length 40
10:06:52.618948 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 262, length 40
10:06:57.607266 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 263, length 40
10:06:57.625225 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 263, length 40
10:06:58.619986 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 264, length 40
10:07:03.594628 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 265, length 40
10:07:03.612588 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 265, length 40
10:07:04.607468 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 266, length 40
10:07:09.596411 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 267, length 40
10:07:09.614100 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 267, length 40
10:07:10.608564 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 268, length 40
10:07:15.597060 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 269, length 40
10:07:15.616658 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 269, length 40
10:07:16.612171 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 270, length 40
10:07:21.596419 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 271, length 40
10:07:21.614464 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 271, length 40
10:07:22.610983 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 272, length 40
10:07:27.606563 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 273, length 40
10:07:27.625479 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 273, length 40
10:07:28.621581 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 274, length 40
10:07:33.597230 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 275, length 40
10:07:33.615464 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 275, length 40
10:07:34.610049 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 276, length 40
10:07:39.592712 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 277, length 40
10:07:39.610250 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 277, length 40
10:07:40.607015 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 278, length 40
10:07:45.589684 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 279, length 40
10:07:45.607893 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 279, length 40
10:07:46.602136 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 280, length 40
10:07:51.591118 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 281, length 40
10:07:51.609105 IP6 lhr48s20-in-x0e.1e100.net > 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx: ICMP6, echo reply, seq 281, length 40
10:07:52.605198 IP6 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx > lhr48s20-in-x0e.1e100.net: ICMP6, echo request, seq 282, length 40

I couldn't see any ICMP6 traffic going over the wrong interface, which is why I just ran tcpdump without -i. From the seq values, you can see an ECHO reply is not present after the previous packet with a reply, so it looks like 50% of the time the ICMP6 packets are disappearing into a blackhole or not being routed correctly.

The latency seen previously seemed to be a temporary issue and not related, as it was normal when testing it again, the 50% loss is the big issue. No idea why it behaves the way on 2.10.x.

aaronjg commented 3 years ago

Just want to check - that tcpdump is from the 2.10.x router? I was actually curious if you could check the tcpdump on the outgoing interface on the 2.8.x interface. I had noticed some situations where the IPv6 packets could go out of the wrong interface on 2.8.x, and made some changes in the 2.10.x release to fix this. So I'm trying to determine if this is a regression in 2.10.x, or if it is just broken in a slightly different way.

Regarding the tcpdump you sared. I'm guessing this defaulted to the br-lan interface without the -i present. And the 2001:470:xxxx:x:xxxx:xxxx:xxxx:xxxx is the client IPv6 address, not the NAT'd source IP address.

Does the same pattern appear when you look at the outgoing WAN interface? I suspect either the even echo requests are not making it out, or the even echo replies are coming in the WAN, but are not making it back to the client.

Also, you can actually do the filtering while capturing using the argument 'host <ipv6adr>)

jamesmacwhite commented 3 years ago

Yes that's correct, the tcpdump was from 2.10.x and you are right it will likely be the client IPv6 address from br-lan.

The same pattern does happen when looking at the specific WAN interface, however for the WAN interface tcpdump you can see a gap in the sequence value, so if the seq value started at 40, you'd next see 42, 44, 46, 48 etc for subsequent pings, there is always a missing gap, comparing that to br-lan, you seem to see the echo request but no echo reply. On the WAN you don't see either, it's missing entirely by the looks of it.

Perhaps it is a regression, if something in 2.8.x is relied on for my setup, unless something is broken with my configuration, that has gone undetected.

aaronjg commented 3 years ago

I see. So every other ping request seems to not make it out of the interface. The thing I am wondering about with 2.8.x is if it was really doing the failover properly, or if it was somehow sending things out of the incorrect interface, and it just appeared to be working.

I also wonder with 2.10.x if the packets are leaving through another interface or are being swallowed by the router. You said that you didn't see any icmp6 traffic over the wrong interface. Did you check each one with with tcpdump -i? Are they being correctly NAT'd so they all have the source IP of the router? If you haven't done that, could you run that diagnostic and report back?

If it does appear that the packets are being swallowed by the router, we need to figure what is going on with the firewall mark. Could you add these rules to the firewall table so we can monitor the marks and then report back?

IP=$(resolveip ipv6.google.com)
ip6tables --table mangle  -I FORWARD 1 -d $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle  -I FORWARD 1 -s $IP -j LOG --log-prefix "_forward start "
ip6tables --table mangle -A FORWARD  -d $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -A FORWARD  -s $IP -j LOG --log-prefix "_forward end "
ip6tables --table mangle -I POSTROUTING 1  -d $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -I POSTROUTING  1 -s $IP -j LOG --log-prefix "postroute start  "
ip6tables --table mangle -A POSTROUTING  -d $IP -j LOG --log-prefix "postroute end  "
ip6tables --table mangle -A POSTROUTING  -s $IP -j LOG --log-prefix "postroute end  "

jamesmacwhite commented 3 years ago

I did check every other IPv6 WAN interface when doing a continuous ping from a LAN client and didn't see any IPv6 ping packets for ipv6.google.com on them, so they were at least going to the primary IPv6 interface I believe, just every other packet dropped. The NAT6 address looked correct from what I could see on the primary WAN.

You could be right on the failover behaviour in 2.8, as I've noticed on multiple occasions that for some reason my second IPv6 interface in the failover policy is suddenly is use when the primary IPv6 interface is reported up and nothing wrong.

I tried removing all members except a single WAN and WAN6 member on the default rules which the ICMP rule would hit and the 50% loss issue still happens.

I can add those rules to my custom firewall. How would you like me to report back the data from those added rules when on 2.10.x and pinging ipv6.google.com?

Might take a bit of time when no one's using the network, I essentially have to keep upgrading and downgrading the mwan3 package with opkg, because I can't keep running 2.10.x for too long given the IPv6 issue I have.

aaronjg commented 3 years ago

Might take a bit of time when no one's using the network, I essentially have to keep upgrading and downgrading the mwan3 package with opkg, because I can't keep running 2.10.x for too long given the IPv6 issue I have.

Fair enough. I'll see if I can reproduce on my end with the details you provided. Your IPV6 setup is still identical to what you have in your medium post? While you are on the 2.8.x version, could you check that the packets are actually all leaving out of the correct interface though?

I suspect there is something weird going on with the NAT66 and mwan3 that is causing this behavior, but I need to investigate more to figure exactly what the issue is.

I can add those rules to my custom firewall. How would you like me to report back the data from those added rules when on 2.10.x and pinging ipv6.google.com?

Once you add the rules and ping, iptables will make an entry in syslog, so you can just read them out with logread and report back.

If packets are going out of the wrong interface in 2.8.x, we can likely debug this on 2.8.x and make the fix on both versions.

jamesmacwhite commented 3 years ago

Fair enough. I'll see if I can reproduce on my end with the details you provided. Your IPV6 setup is still identical to what you have in your medium post? While you are on the 2.8.x version, could you check that the packets are actually all leaving out of the correct interface though?

They seem to be. Doing a ping from a LAN client and running tcpdump on the primary interface I see the echo request and replies in sequence without any gaps, so not going across any other interface other than the primary.

This is probably better for the most up to date setup I am using: https://gist.github.com/jamesmacwhite/58757c67cf6566c3d6cff46ece2fea32

Once you add the rules and ping, iptables will make an entry in syslog, so you can just read them out with logread and report back.

I'm not seeing any log entries, doing ip6tables -vnL -t mangle I can see the rules in the FORWARD and POSTROUTING chain aren't matching as the pkts value is zero.

aaronjg commented 3 years ago

This is probably better for the most up to date setup I am using

Most of my configuration is failover with 3 physical WAN connections

Sorry, a little confused. That link looks like the IPv6 interfaces are virtual interfaces for the first two for ipv6. For wan_wanb_wanc (aaisp6, henet). Am I reading that correctly?

A bit surprised that those rules are not being hit. The -I POSTROUTING 1 rule should go in at the top of the postrouting chain, so should be evaluated for all the traffic leaving the router.

https://i.stack.imgur.com/68Cvx.png

Maybe there is some round robin DNS and the IP addresses aren't matching?

jamesmacwhite commented 3 years ago

Yeah, you are absolutely right. It is likely due to ipv6.google.com having high availability, I'll use one of the specific IP addresses directly from a DNS lookup to avoid the DNS round robin issue and lock that in the firewall rules. That's worked I can now see the logging happening.

2607:f8b0:4002:c00::66 is ipv6.google.com in the log output.

My inferface setup is a bit confusing, aaisp6 is a virtual interface of aaisp and henet is a 6in4 tunnel not a virtual interface. They are then used as policies like this. I have the wan_wanb_wanc policy as default.

config policy 'wan_wanb_wanc'
    list use_member 'wan_m1_w3'
    list use_member 'wanb_m2_w2'
    list use_member 'wanc_m3_w1'
    list use_member 'aaisp6_m1_w3'
    list use_member 'henet_m2_w2'
    list use_member 'wanc6_m3_w1'
    option last_resort 'unreachable'

They don't follow the naming convention of wan6, wanb6, because technically they aren't and are coming from that provider and from somewhere else, so the wan6, wanb6 interface are left disabled for now until those providers actually deploy IPv6. If however, it makes more sense to do that, I could change it.

What could be an issue is I noticed that briefly, the SRC IP of the Windows client was reporting being from the HE.net prefix which is not the primary, but going out from the L2TP interface which is wrong:

2001:0470:xxxx - HE.net /48 prefix which should be 6in4-henet. However we see l2tp-aaisp. So that looks suspect.

At the moment my DHCPv6 server is serving both /48 prefixes across the LAN, I wonder if Windows selecting the source address is also related?

Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.287021] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3643 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.311569] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3643 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.335944] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3643 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.356123] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3643 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.398537] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3644 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.423082] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3644 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.447451] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3644 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.467627] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3644 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.508933] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3645 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.533473] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3645 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.557841] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3645 MARK=0x800
Mon Dec 28 08:36:55 2020 kern.warn kernel: [1629750.578017] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=1 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3645 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.528333] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3646 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.552884] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3646 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.577269] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3646 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.597451] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3646 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.640275] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3647 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.664826] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3647 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.689203] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3647 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.709379] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3647 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.754864] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3648 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.779425] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3648 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.803809] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3648 MARK=0x800
Mon Dec 28 08:36:56 2020 kern.warn kernel: [1629751.823987] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=2 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3648 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.771844] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3649 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.796394] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3649 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.820778] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3649 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.840953] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3649 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.886414] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3650 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.910977] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3650 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.935367] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3650 MARK=0x800
Mon Dec 28 08:36:57 2020 kern.warn kernel: [1629752.955560] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3650 MARK=0x800
Mon Dec 28 08:36:58 2020 kern.warn kernel: [1629753.002182] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3651 MARK=0x800
Mon Dec 28 08:36:58 2020 kern.warn kernel: [1629753.026733] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3651 MARK=0x800
Mon Dec 28 08:36:58 2020 kern.warn kernel: [1629753.051113] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3651 MARK=0x800
Mon Dec 28 08:36:58 2020 kern.warn kernel: [1629753.071305] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=3 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=3651 MARK=0x800

Interestingly, now the IPv6 traffic is going over 6in4-henet even though it's not the first policy

Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.376025] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4294 MARK=0x800
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.400656] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4294 MARK=0x800
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.425120] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4294 MARK=0x800
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.445382] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4294 MARK=0x800
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.575402] _forward start IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4294 MARK=0x3f00
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.596539] _forward end IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4294 MARK=0x3f00
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.617505] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4294 MARK=0x3f00
Mon Dec 28 08:49:37 2020 kern.warn kernel: [1630512.637507] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4294 MARK=0x3f00
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.391072] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4295 MARK=0x800
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.415703] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4295 MARK=0x800
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.440161] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4295 MARK=0x800
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.460427] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4295 MARK=0x800
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.590054] _forward start IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4295 MARK=0x3f00
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.611192] _forward end IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4295 MARK=0x3f00
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.632164] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4295 MARK=0x3f00
Mon Dec 28 08:49:38 2020 kern.warn kernel: [1630513.652164] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4295 MARK=0x3f00
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.405835] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4296 MARK=0x800
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.430466] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4296 MARK=0x800
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.454925] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4296 MARK=0x800
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.475190] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4296 MARK=0x800
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.604859] _forward start IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4296 MARK=0x3f00
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.625998] _forward end IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4296 MARK=0x3f00
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.646972] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4296 MARK=0x3f00
Mon Dec 28 08:49:39 2020 kern.warn kernel: [1630514.666975] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4296 MARK=0x3f00
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.419263] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4297 MARK=0x800
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.443898] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4297 MARK=0x800
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.468355] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4297 MARK=0x800
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.488622] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:e52c:8236:47fe:f0a1 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=4297 MARK=0x800
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.660031] _forward start IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4297 MARK=0x3f00
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.681173] _forward end IN=6in4-henet OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4297 MARK=0x3f00
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.702148] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4297 MARK=0x3f00
Mon Dec 28 08:49:40 2020 kern.warn kernel: [1630515.722151] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:e52c:8236:47fe:f0a1 LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=4297 MARK=0x3f00

I wonder if having multiple prefixes with NAT6 is a bad idea?

aaronjg commented 3 years ago

What could be an issue is I noticed that briefly, the SRC IP of the Windows client was reporting being from the HE.net prefix which is not the primary, but going out from the L2TP interface which is wrong:

Thanks for the diagnostics. Yes, this is the issue, but it's not an issue with your setup. It shouldn't matter what the client prefix is, since the router will just be masquerading it anyway.

It was, however, due to a regression that was introduced in the 2.10.x factor. Sorry for the inconvenience, and thank you for the testing and locating the bug!

This branch should fix the issue for you. https://github.com/aaronjg/openwrt-packages/tree/bugfix/mwan3-ipv6-regression

jamesmacwhite commented 3 years ago

Thank you! I will compile the mwan3 package from your branch to test it out and report back!

jamesmacwhite commented 3 years ago

I'm afraid I'm still seeing the same 50% loss issue under your branch with the same problem. The inbound and outbound interfaces do seem to be rotating around each time.

I compiled your branch with the SDK and installed the generated package locally.

Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.008960] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7358 MARK=0x800
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.033591] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7358 MARK=0x800
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.058054] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7358 MARK=0x800
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.078317] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7358 MARK=0x800
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.205764] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7358 MARK=0x3f00
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.226908] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7358 MARK=0x3f00
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.247883] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7358 MARK=0x3f00
Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.267889] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7358 MARK=0x3f00
Tue Dec 29 07:53:57 2020 kern.warn kernel: [1713601.025831] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7359 MARK=0x3f00
Tue Dec 29 07:53:57 2020 kern.warn kernel: [1713601.050550] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7359 MARK=0x3f00
Tue Dec 29 07:53:57 2020 kern.warn kernel: [1713601.075098] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7359 MARK=0x3f00
Tue Dec 29 07:53:57 2020 kern.warn kernel: [1713601.095448] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7359 MARK=0x3f00
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713605.913505] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7360 MARK=0x800
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713605.938138] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7360 MARK=0x800
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713605.962599] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7360 MARK=0x800
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713605.982861] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7360 MARK=0x800
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713606.109549] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7360 MARK=0x3f00
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713606.130688] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7360 MARK=0x3f00
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713606.151658] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7360 MARK=0x3f00
Tue Dec 29 07:54:02 2020 kern.warn kernel: [1713606.171657] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7360 MARK=0x3f00
Tue Dec 29 07:54:03 2020 kern.warn kernel: [1713606.928813] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7361 MARK=0x3f00
Tue Dec 29 07:54:03 2020 kern.warn kernel: [1713606.953532] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7361 MARK=0x3f00
Tue Dec 29 07:54:03 2020 kern.warn kernel: [1713606.978080] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7361 MARK=0x3f00
Tue Dec 29 07:54:03 2020 kern.warn kernel: [1713606.998428] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7361 MARK=0x3f00
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713611.921466] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7362 MARK=0x800
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713611.946097] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7362 MARK=0x800
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713611.970561] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7362 MARK=0x800
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713611.990824] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7362 MARK=0x800
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713612.118215] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7362 MARK=0x3f00
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713612.139352] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7362 MARK=0x3f00
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713612.160318] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7362 MARK=0x3f00
Tue Dec 29 07:54:08 2020 kern.warn kernel: [1713612.180322] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7362 MARK=0x3f00
Tue Dec 29 07:54:09 2020 kern.warn kernel: [1713612.938471] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7363 MARK=0x3f00
Tue Dec 29 07:54:09 2020 kern.warn kernel: [1713612.963191] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7363 MARK=0x3f00
Tue Dec 29 07:54:09 2020 kern.warn kernel: [1713612.987741] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7363 MARK=0x3f00
Tue Dec 29 07:54:09 2020 kern.warn kernel: [1713613.008093] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7363 MARK=0x3f00
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713617.918249] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7364 MARK=0x800
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713617.942879] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7364 MARK=0x800
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713617.967340] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7364 MARK=0x800
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713617.987604] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7364 MARK=0x800
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713618.122808] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7364 MARK=0x3f00
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713618.143946] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7364 MARK=0x3f00
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713618.164913] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7364 MARK=0x3f00
Tue Dec 29 07:54:14 2020 kern.warn kernel: [1713618.184917] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7364 MARK=0x3f00
Tue Dec 29 07:54:15 2020 kern.warn kernel: [1713618.931111] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7365 MARK=0x3f00
Tue Dec 29 07:54:15 2020 kern.warn kernel: [1713618.955827] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7365 MARK=0x3f00
Tue Dec 29 07:54:15 2020 kern.warn kernel: [1713618.980372] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7365 MARK=0x3f00
Tue Dec 29 07:54:15 2020 kern.warn kernel: [1713619.000727] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7365 MARK=0x3f00
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713623.925712] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7366 MARK=0x800
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713623.950344] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7366 MARK=0x800
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713623.974800] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7366 MARK=0x800
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713623.995065] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7366 MARK=0x800
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713624.122031] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7366 MARK=0x3f00
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713624.143170] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7366 MARK=0x3f00
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713624.164137] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7366 MARK=0x3f00
Tue Dec 29 07:54:20 2020 kern.warn kernel: [1713624.184139] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7366 MARK=0x3f00
Tue Dec 29 07:54:21 2020 kern.warn kernel: [1713624.940167] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7367 MARK=0x3f00
Tue Dec 29 07:54:21 2020 kern.warn kernel: [1713624.964885] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7367 MARK=0x3f00
Tue Dec 29 07:54:21 2020 kern.warn kernel: [1713624.989426] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7367 MARK=0x3f00
Tue Dec 29 07:54:21 2020 kern.warn kernel: [1713625.009778] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7367 MARK=0x3f00
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713629.914609] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7368 MARK=0x800
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713629.939247] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7368 MARK=0x800
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713629.963715] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7368 MARK=0x800
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713629.983981] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7368 MARK=0x800
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713630.111009] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7368 MARK=0x3f00
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713630.132149] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7368 MARK=0x3f00
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713630.153119] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7368 MARK=0x3f00
Tue Dec 29 07:54:26 2020 kern.warn kernel: [1713630.173122] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7368 MARK=0x3f00
Tue Dec 29 07:54:27 2020 kern.warn kernel: [1713630.926797] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7369 MARK=0x3f00
Tue Dec 29 07:54:27 2020 kern.warn kernel: [1713630.951516] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7369 MARK=0x3f00
Tue Dec 29 07:54:27 2020 kern.warn kernel: [1713630.976062] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7369 MARK=0x3f00
Tue Dec 29 07:54:27 2020 kern.warn kernel: [1713630.996415] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7369 MARK=0x3f00
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713635.917080] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7370 MARK=0x800
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713635.941710] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7370 MARK=0x800
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713635.966173] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7370 MARK=0x800
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713635.986437] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7370 MARK=0x800
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713636.121836] _forward start IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7370 MARK=0x3f00
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713636.142975] _forward end IN=l2tp-aaisp OUT=br-lan MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7370 MARK=0x3f00
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713636.163942] postroute start  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7370 MARK=0x3f00
Tue Dec 29 07:54:32 2020 kern.warn kernel: [1713636.183943] postroute end  IN= OUT=br-lan SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a LEN=80 TC=0 HOPLIMIT=107 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7370 MARK=0x3f00
Tue Dec 29 07:54:33 2020 kern.warn kernel: [1713636.931204] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7371 MARK=0x3f00
Tue Dec 29 07:54:33 2020 kern.warn kernel: [1713636.955922] _forward end IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7371 MARK=0x3f00
Tue Dec 29 07:54:33 2020 kern.warn kernel: [1713636.980466] postroute start  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7371 MARK=0x3f00
Tue Dec 29 07:54:33 2020 kern.warn kernel: [1713637.000820] postroute end  IN= OUT=6in4-henet SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7371 MARK=0x3f00

aaronjg commented 3 years ago

Thanks. No need to recompile for this since it is just script updates and no changes to the compiled library.

There must have been multiple issues then. This log at least more clearly shows the issue. For example the two packets below have identical IN interface, SRC address, and DST address, yet they are being marked differently by iptables.

Tue Dec 29 07:53:56 2020 kern.warn kernel: [1713600.008960] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7358 MARK=0x800
...
Tue Dec 29 07:53:57 2020 kern.warn kernel: [1713601.025831] _forward start IN=br-lan OUT=6in4-henet MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=127 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7359 MARK=0x3f00

You had mentioned that the Windows machine was sending packets with different src ips, but I'm not seeing this in the log you shared. Was this in a different log? It does seem like that could be related to the issue...

I'm going to see if I can reproduce your set up to recreate this issue. Seems like we are getting closer at least. Thanks for the help with the logs.

jamesmacwhite commented 3 years ago

I just compiled instead of a copying the files in case I missed them in the paths just to be sure.

The Windows src address theory was because I have multiple /48 prefixes handed out, essentially the Windows machine will have one or more IPv6 addresses configured:

A ULA address from the defined ULA prefix
A global IPv6 address from l2tp-aaisp from the /48
A global IPv6 address from 6in4-henet from the /48

The ULA won't be a problem, because by default most clients will go for a global IPv6 address. For NAT6 to work technically you need a global prefix to be announced otherwise clients would prefer IPv4 anyway.

What I did wonder is, under 2.8.x it seems that it was using the HE.net prefix address, but going out over the l2tp-aaisp interface, which that prefix doesn't belong to. For example, right now if do a traceroute to an IPv6 address from a Windows client.

The below logs are all from 2.8.x branch

C:\Users\james>tracert -6 2607:f8b0:4002:c00::66

Tracing route to 2607:f8b0:4002:c00::66 over a maximum of 30 hops

  1     2 ms     3 ms     2 ms  linksys-wrt3200acm.internal.jmwhite.co.uk [2001:470:6839::1]
  2   118 ms   109 ms   113 ms  careless.aa.net.uk [2001:8b0:0:53::5a9b:3509]
  3   110 ms   111 ms   114 ms  o.aimless.tch.aa.net.uk [2001:8b0:0:53::105]
  4   109 ms   110 ms   109 ms  2001:4860:1:1::1592
  5   109 ms     *        *     2001:4860:0:1101::f
  6   207 ms   199 ms   198 ms  2607:f8b0:e000:8000::6
  7   201 ms   204 ms   199 ms  2001:4860::c:4001:9faf
  8   199 ms   198 ms   200 ms  2001:4860::c:4001:5890
  9   200 ms   207 ms   200 ms  2001:4860::c:4001:557b
 10   198 ms   199 ms   200 ms  2001:4860::cc:4002:d6
 11     *        *        *     Request timed out.
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14     *        *        *     Request timed out.
 15     *        *        *     Request timed out.

We can see the traffic is clearly traversing the AAISP network, but the iptables logging shows that the SRC address was the from the 2001:470:6839::/48 prefix, which is HE.net not AAISP. My router has a ::1 address from each prefix too and it seems in this case it decided to use the HE.net ::1 address as the first hop.

So I wondered if this had something to do with it. Technically with NAT6 it shouldn't, because the outgoing IPv6 shows as AAISP with various sites and traceroute confirms the traffic path, but it's something else to consider in the puzzle.

Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.046993] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7637 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.071536] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7637 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.095914] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7637 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.116089] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7637 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.250896] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7638 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.275441] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7638 MARK=0x800
Tue Dec 29 14:29:57 2020 kern.warn kernel: [1737369.299819] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7638 MARK=0x800
Tue Dec 29 14:29:58 2020 kern.warn kernel: [1737369.319996] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7638 MARK=0x800
Tue Dec 29 14:29:58 2020 kern.warn kernel: [1737369.451458] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7639 MARK=0x800
Tue Dec 29 14:29:58 2020 kern.warn kernel: [1737369.476001] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7639 MARK=0x800
Tue Dec 29 14:29:58 2020 kern.warn kernel: [1737369.500369] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7639 MARK=0x800
Tue Dec 29 14:29:58 2020 kern.warn kernel: [1737369.520547] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=9 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7639 MARK=0x800
Tue Dec 29 14:30:00 2020 kern.warn kernel: [1737371.704825] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7640 MARK=0x800
Tue Dec 29 14:30:00 2020 kern.warn kernel: [1737371.729454] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7640 MARK=0x800
Tue Dec 29 14:30:00 2020 kern.warn kernel: [1737371.753913] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7640 MARK=0x800
Tue Dec 29 14:30:00 2020 kern.warn kernel: [1737371.774177] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7640 MARK=0x800
Tue Dec 29 14:30:04 2020 kern.warn kernel: [1737375.684480] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7641 MARK=0x800
Tue Dec 29 14:30:04 2020 kern.warn kernel: [1737375.709111] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7641 MARK=0x800
Tue Dec 29 14:30:04 2020 kern.warn kernel: [1737375.733573] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7641 MARK=0x800
Tue Dec 29 14:30:04 2020 kern.warn kernel: [1737375.753837] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7641 MARK=0x800
Tue Dec 29 14:30:08 2020 kern.warn kernel: [1737379.677547] _forward start IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7642 MARK=0x800
Tue Dec 29 14:30:08 2020 kern.warn kernel: [1737379.702178] _forward end IN=br-lan OUT=l2tp-aaisp MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7642 MARK=0x800
Tue Dec 29 14:30:08 2020 kern.warn kernel: [1737379.726642] postroute start  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7642 MARK=0x800
Tue Dec 29 14:30:08 2020 kern.warn kernel: [1737379.746907] postroute end  IN= OUT=l2tp-aaisp SRC=2001:0470:6839:0000:11bf:4b6e:8ad2:fe6a DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=112 TC=0 HOPLIMIT=10 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7642 MARK=0x800

I did notice the mark values, so perhaps this is where the problem really is.

jamesmacwhite commented 3 years ago

I guess there are a few issues based on the logs and comparing 2.8.x to 2.10.x, summarising the issues found.

There's potentially an issue with the mark values as shown in the 2.10.x log shown here: https://github.com/openwrt/packages/issues/14332#issuecomment-751986335
There are multiple outgoing interfaces involved when the failover configuration policy should be l2tp-aaisp, then 6in4-hetnet if l2tp-aaisp is down, so there is no reason to see 6in4-henet at all I don't think, but it does explain why there's a 50% ping loss, when it seems the ICMP6 packets are alternating between two interfaces.
Are there potentially problems with multiple prefixes from different WANs and clients such as Windows just using whatever it wants? I guess doing the same ping test but locking to a specific source address might answer that question.

I don't know if others are having IPv6 issues with 2.10.x or if it's localised to my specific setup. I guess it is hard to know given we think a lot of mwan3 usage is still IPv4 only.

jamesmacwhite commented 3 years ago

@aaronjg Would it help if I went back through the package versions and tested from around 2.9.0 onwards, maybe this can help isolate the specific commit that is perhaps causing the regression?

From memory, I believe early 2.9.x versions worked, then around 2.10.x onwards IPv6 was breaking for me.

aaronjg commented 3 years ago

The next diagnostic step is to insert iptables logging rules to what is going on that causes the even packets to be marked with 0x3f00.

I was going to set up something to see if I could replicate, but if you can add the logging rules, that would be great!

jamesmacwhite commented 3 years ago

Happy to assist debugging, let me know what iptables logging rules you want me to run to help with that.

I did build a few old versions of 2.10.x but they seemed to have other issues, so probably not helpful for testing this, I guess we'll have to narrow it down through more logs.

0x3F00 being the default firewall mask of mwan3, although under 2.8.x the ICMP packets are all 0x800 but for some reason on 2.10.x they seem to be alternating between 0x800 and 0x3F00 which somewhat explains why the 50% loss issue is present, but why they are marked that way, I guess is the main issue to resolve.

Let me know what iptables logging rules would be beneficial and I'll temporarily upgrade the mwan3 package and run a ICMP ping test.

aaronjg commented 3 years ago

Try these rules - these should track the packet through the mwan3 hook rules. After that, may need to dig into the NAT table or some of the mwan3 chains, depending on what these results show.

IP=<IPV6>
ip6tables --table mangle -A mwan3_hook -d $IP  -j LOG --log-prefix "rule end "
ip6tables --table mangle -A mwan3_hook -s $IP  -j LOG --log-prefix "rule end "
ip6tables --table mangle -I mwan3_hook 11 -d $IP  -j LOG --log-prefix "after conmark save "
ip6tables --table mangle -I mwan3_hook 11 -s $IP  -j LOG --log-prefix "after conmark save "
ip6tables --table mangle -I mwan3_hook 10 -d $IP  -j LOG --log-prefix "after mwan3_rules "
ip6tables --table mangle -I mwan3_hook 10 -s $IP  -j LOG --log-prefix "after mwan3_rules "
ip6tables --table mangle -I mwan3_hook 9 -d $IP  -j LOG --log-prefix "after mwan3_connected "
ip6tables --table mangle -I mwan3_hook 9 -s $IP  -j LOG --log-prefix "after mwan3_connected "
ip6tables --table mangle -I mwan3_hook 8 -d $IP  -j LOG --log-prefix "after mwan3_ifaces_in "
ip6tables --table mangle -I mwan3_hook 8 -s $IP  -j LOG --log-prefix "after mwan3_ifaces_in "
ip6tables --table mangle -I mwan3_hook 7 -d $IP  -j LOG --log-prefix "after after conmark restore "
ip6tables --table mangle -I mwan3_hook 7 -s $IP  -j LOG --log-prefix "after after conmark restore "
ip6tables --table mangle -I mwan3_hook 6  -d $IP  -j LOG --log-prefix "rule start "
ip6tables --table mangle -I mwan3_hook 6  -s $IP  -j LOG --log-prefix "rule start "

#check rules
ip6tables -t mangle -S mwan3_hook

jamesmacwhite commented 3 years ago

Thanks, just to confirm though, shouldn't this be for ip6tables?

aaronjg commented 3 years ago

Thanks, just to confirm though, shouldn't this be for ip6tables?

Yes.

jamesmacwhite commented 3 years ago

Thought so, what is the $RULE variable meant to be though?

aaronjg commented 3 years ago

You can take that part out. It was from a previous debugging script I copied.

jamesmacwhite commented 3 years ago

OK thanks for clarifying. This is what ip6tables -t mangle -S mwan3_hook shows with those custom rules added.

-N mwan3_hook
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule start "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule start "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 133 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after after conmark restore "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after after conmark restore "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_ifaces_in "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_ifaces_in "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_connected "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_connected "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_rules "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_rules "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 137 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after conmark save "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after conmark save "
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_ifaces_in
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark ! --mark 0x3f00/0x3f00 -j mwan3_connected
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule end "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule end "
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule end "

Fri Jan  1 21:00:30 2021 kern.warn kernel: [2020097.593568] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:30 2021 kern.warn kernel: [2020097.616022] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:30 2021 kern.warn kernel: [2020097.639954] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:30 2021 kern.warn kernel: [2020097.663360] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:30 2021 kern.warn kernel: [2020097.686765] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:31 2021 kern.warn kernel: [2020097.709820] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948
Fri Jan  1 21:00:31 2021 kern.warn kernel: [2020097.732963] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948 MARK=0x3f00
Fri Jan  1 21:00:31 2021 kern.warn kernel: [2020097.756279] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7948 MARK=0x3f00
Fri Jan  1 21:00:35 2021 kern.warn kernel: [2020102.592570] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:35 2021 kern.warn kernel: [2020102.615022] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:35 2021 kern.warn kernel: [2020102.638952] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:35 2021 kern.warn kernel: [2020102.662357] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:35 2021 kern.warn kernel: [2020102.685761] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.708818] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.731985] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949 MARK=0x800
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.755215] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7949 MARK=0x800
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.886920] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.907186] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.928931] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.950154] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.971376] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.992249] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020103.013211] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020103.606031] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020103.628481] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020103.652413] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:37 2021 kern.warn kernel: [2020103.675820] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:37 2021 kern.warn kernel: [2020103.699225] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:37 2021 kern.warn kernel: [2020103.722281] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950
Fri Jan  1 21:00:37 2021 kern.warn kernel: [2020103.745424] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950 MARK=0x3f00
Fri Jan  1 21:00:37 2021 kern.warn kernel: [2020103.768741] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7950 MARK=0x3f00
Fri Jan  1 21:00:41 2021 kern.warn kernel: [2020108.584702] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:41 2021 kern.warn kernel: [2020108.607152] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:41 2021 kern.warn kernel: [2020108.631079] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:41 2021 kern.warn kernel: [2020108.654485] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:41 2021 kern.warn kernel: [2020108.677887] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.700942] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.724102] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951 MARK=0x800
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.747330] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7951 MARK=0x800
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.877025] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.897289] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.919033] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.940255] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.961478] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020108.982350] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.003308] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7951 MARK=0x3f00
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.598787] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.621236] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.645169] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.668575] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:42 2021 kern.warn kernel: [2020109.691979] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:43 2021 kern.warn kernel: [2020109.715035] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952
Fri Jan  1 21:00:43 2021 kern.warn kernel: [2020109.738178] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952 MARK=0x3f00
Fri Jan  1 21:00:43 2021 kern.warn kernel: [2020109.761494] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7952 MARK=0x3f00
Fri Jan  1 21:00:47 2021 kern.warn kernel: [2020114.589887] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:47 2021 kern.warn kernel: [2020114.612338] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:47 2021 kern.warn kernel: [2020114.636267] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:47 2021 kern.warn kernel: [2020114.659673] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:47 2021 kern.warn kernel: [2020114.683081] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.706139] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.729312] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953 MARK=0x800
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.752543] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7953 MARK=0x800
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.883865] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.904134] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.925884] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.947107] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.968331] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020114.989204] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.010165] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7953 MARK=0x3f00
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.603499] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.625954] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.649888] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.673296] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:48 2021 kern.warn kernel: [2020115.696700] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:49 2021 kern.warn kernel: [2020115.719756] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954
Fri Jan  1 21:00:49 2021 kern.warn kernel: [2020115.742899] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954 MARK=0x3f00
Fri Jan  1 21:00:49 2021 kern.warn kernel: [2020115.766216] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7954 MARK=0x3f00
Fri Jan  1 21:00:53 2021 kern.warn kernel: [2020120.593633] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:53 2021 kern.warn kernel: [2020120.616082] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:53 2021 kern.warn kernel: [2020120.640012] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:53 2021 kern.warn kernel: [2020120.663421] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:53 2021 kern.warn kernel: [2020120.686842] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.709901] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.733077] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955 MARK=0x800
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.756307] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7955 MARK=0x800
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.894345] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.914621] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.936371] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.957596] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.978820] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020120.999694] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.020656] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7955 MARK=0x3f00
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.604908] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.627362] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.651293] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.674700] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:54 2021 kern.warn kernel: [2020121.698103] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:55 2021 kern.warn kernel: [2020121.721158] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956
Fri Jan  1 21:00:55 2021 kern.warn kernel: [2020121.744301] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956 MARK=0x3f00
Fri Jan  1 21:00:55 2021 kern.warn kernel: [2020121.767618] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7956 MARK=0x3f00
Fri Jan  1 21:00:59 2021 kern.warn kernel: [2020126.594621] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:00:59 2021 kern.warn kernel: [2020126.617077] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:00:59 2021 kern.warn kernel: [2020126.641009] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:00:59 2021 kern.warn kernel: [2020126.664434] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:00:59 2021 kern.warn kernel: [2020126.687847] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.710906] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.734082] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957 MARK=0x800
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.757314] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7957 MARK=0x800
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.896524] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.916796] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.938547] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.959772] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020126.981013] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020127.001894] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020127.022859] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7957 MARK=0x3f00
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020127.610990] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020127.633482] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:00 2021 kern.warn kernel: [2020127.657443] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:01 2021 kern.warn kernel: [2020127.680881] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:01 2021 kern.warn kernel: [2020127.704308] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:01 2021 kern.warn kernel: [2020127.727384] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958
Fri Jan  1 21:01:01 2021 kern.warn kernel: [2020127.750549] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958 MARK=0x3f00
Fri Jan  1 21:01:01 2021 kern.warn kernel: [2020127.773870] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7958 MARK=0x3f00

aaronjg commented 3 years ago

Ah shoot. I forgot about the icmp6 rules on the ipv6. I think you need to add 5 to the rule insert locations so they go in the correct place.

aaronjg commented 3 years ago

Do you have any other rules that are marking the packets?

This line looks suspicious to me. It is a ping return, but at 'rule start' it should be before the first mwan3 rule is applied, but it already has a mark of 0x3f00.

Fri Jan  1 21:00:36 2021 kern.warn kernel: [2020102.886920] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7949 MARK=0x3f00

jamesmacwhite commented 3 years ago

There shouldn't be any rules matching the destination IP being pinged other than the default v6 rule in mwan3. I do have ICMP IPv6 related rules in mwan3 but they would not be targeting this IP range and would be for a different WAN policy.

Here's the same from 2.8.x for a comparision:

Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.353863] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.376315] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.400246] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.423655] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.447062] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.470118] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.493289] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7998 MARK=0x800
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.624788] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.645053] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.666798] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.688019] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.709241] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.730114] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x3f00
Fri Jan  1 22:19:17 2021 kern.warn kernel: [2024825.751082] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7998 MARK=0x800
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.369851] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.392305] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.416235] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.439641] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.463044] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.486101] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.509255] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=7999 MARK=0x800
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.638705] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.658974] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.680721] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.701947] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.723170] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.744042] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x3f00
Fri Jan  1 22:19:18 2021 kern.warn kernel: [2024826.765010] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=7999 MARK=0x800
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.382679] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.405133] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.429062] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.452468] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.475873] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.498929] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.522082] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8000 MARK=0x800
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.651537] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.671803] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.693548] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.714770] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.735990] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.756862] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x3f00
Fri Jan  1 22:19:19 2021 kern.warn kernel: [2024827.777828] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8000 MARK=0x800
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.398128] rule start IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.420578] after after conmark restore IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.444513] after mwan3_ifaces_in IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.467921] after mwan3_connected IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.491325] after mwan3_rules IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.514382] after conmark save IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.537532] rule end IN=br-lan OUT= MAC=32:23:03:df:2c:80:04:d4:c4:4f:3b:bf:86:dd SRC=2001:0470:6839:0000:84ac:495e:f221:3619 DST=2607:f8b0:4002:0c00:0000:0000:0000:0066 LEN=80 TC=0 HOPLIMIT=128 FLOWLBL=0 PROTO=ICMPv6 TYPE=128 CODE=0 ID=1 SEQ=8001 MARK=0x800
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.668480] rule start IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.688764] after after conmark restore IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.710513] after mwan3_ifaces_in IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.731738] after mwan3_connected IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.752963] after mwan3_rules IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.773837] after conmark save IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x3f00
Fri Jan  1 22:19:20 2021 kern.warn kernel: [2024828.794809] rule end IN=l2tp-aaisp OUT= MAC= SRC=2607:f8b0:4002:0c00:0000:0000:0000:0066 DST=2001:08b0:1111:1111:0000:ffff:51bb:edf2 LEN=80 TC=0 HOPLIMIT=109 FLOWLBL=0 PROTO=ICMPv6 TYPE=129 CODE=0 ID=1 SEQ=8001 MARK=0x800

aaronjg commented 3 years ago

Okay, so there is something going on in your setup with connmark. Could you share your full mangle and nat tables with ip6tables -t nat -S and `ip6tables -t mangle -S'.

I think the problem is here: https://github.com/openwrt/packages/blob/master/net/mwan3/files/lib/mwan3/mwan3.sh#L296-L298

If you remove line 297 -m mark --mark 0x0/$MMX_MASK \ it may fix your issue, but I'd still like to understand how the packets are getting marked in the first place.

jamesmacwhite commented 3 years ago

Sure this is from 2.8.x, I can do it from 2.10.x if you want as well.

root@linksys-wrt3200acm:~# ip6tables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N postrouting_guest_rule
-N postrouting_lan_rule
-N postrouting_rule
-N postrouting_vpn_rule
-N postrouting_wan_rule
-N postrouting_wireguard_rule
-N prerouting_guest_rule
-N prerouting_lan_rule
-N prerouting_rule
-N prerouting_vpn_rule
-N prerouting_wan_rule
-N prerouting_wireguard_rule
-N zone_guest_postrouting
-N zone_guest_prerouting
-N zone_lan_postrouting
-N zone_lan_prerouting
-N zone_vpn_postrouting
-N zone_vpn_prerouting
-N zone_wan_postrouting
-N zone_wan_prerouting
-N zone_wireguard_postrouting
-N zone_wireguard_prerouting
-A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
-A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i l2tp-aaisp -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i 6in4-henet -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth0.4 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth1.2 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i pppoe-wanb -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth0.3 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i wlan1-1 -m comment --comment "!fw3" -j zone_guest_prerouting
-A PREROUTING -i tun0 -m comment --comment "!fw3" -j zone_vpn_prerouting
-A PREROUTING -i wg -m comment --comment "!fw3" -j zone_wireguard_prerouting
-A PREROUTING -i wgb -m comment --comment "!fw3" -j zone_wireguard_prerouting
-A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
-A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o l2tp-aaisp -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o 6in4-henet -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth0.4 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth1.2 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o pppoe-wanb -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth0.3 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o wlan1-1 -m comment --comment "!fw3" -j zone_guest_postrouting
-A POSTROUTING -o tun0 -m comment --comment "!fw3" -j zone_vpn_postrouting
-A POSTROUTING -o wg -m comment --comment "!fw3" -j zone_wireguard_postrouting
-A POSTROUTING -o wgb -m comment --comment "!fw3" -j zone_wireguard_postrouting
-A zone_guest_postrouting -m comment --comment "!fw3: Custom guest postrouting rule chain" -j postrouting_guest_rule
-A zone_guest_prerouting -m comment --comment "!fw3: Custom guest prerouting rule chain" -j prerouting_guest_rule
-A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
-A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
-A zone_vpn_postrouting -m comment --comment "!fw3: Custom vpn postrouting rule chain" -j postrouting_vpn_rule
-A zone_vpn_prerouting -m comment --comment "!fw3: Custom vpn prerouting rule chain" -j prerouting_vpn_rule
-A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
-A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
-A zone_wireguard_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wireguard_postrouting -m comment --comment "!fw3: Custom wireguard postrouting rule chain" -j postrouting_wireguard_rule
-A zone_wireguard_prerouting -m comment --comment "!fw3: Custom wireguard prerouting rule chain" -j prerouting_wireguard_rule

root@linksys-wrt3200acm:~# ip6tables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N mwan3_connected
-N mwan3_hook
-N mwan3_iface_in_aaisp6
-N mwan3_iface_in_henet
-N mwan3_iface_in_wanc6
-N mwan3_iface_in_wg6
-N mwan3_iface_in_wgb6
-N mwan3_ifaces_in
-N mwan3_policy_aaisp_only
-N mwan3_policy_henet_only
-N mwan3_policy_wan_only
-N mwan3_policy_wan_wanb
-N mwan3_policy_wan_wanb_wanc
-N mwan3_policy_wanb_only
-N mwan3_policy_wanb_wan
-N mwan3_policy_wanc_only
-N mwan3_policy_wg_balanced
-N mwan3_policy_wg_only
-N mwan3_policy_wgb_only
-N mwan3_rule_https
-N mwan3_rules
-A PREROUTING -j mwan3_hook
-A FORWARD -o l2tp-aaisp -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i l2tp-aaisp -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o 6in4-henet -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i 6in4-henet -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth1.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth1.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o pppoe-wanb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wanb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o wg -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i wg -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o wgb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i wgb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A OUTPUT -j mwan3_hook
-A mwan3_connected -m set --match-set mwan3_connected dst -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule start "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule start "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 133 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after after conmark restore "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after after conmark restore "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_ifaces_in "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_ifaces_in "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_connected "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_connected "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_rules "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after mwan3_rules "
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 137 -j RETURN
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after conmark save "
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "after conmark save "
-A mwan3_hook -p ipv6-icmp -m set --match-set mwan3_source_v6 src -m icmp6 --icmpv6-type 128 -j RETURN
-A mwan3_hook -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_ifaces_in
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark ! --mark 0x3f00/0x3f00 -j mwan3_connected
-A mwan3_hook -d 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule end "
-A mwan3_hook -s 2607:f8b0:4002:c00::66/128 -j LOG --log-prefix "rule end "
-A mwan3_iface_in_aaisp6 -i l2tp-aaisp -m set --match-set mwan3_connected_v6 src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_aaisp6 -i l2tp-aaisp -m mark --mark 0x0/0x3f00 -m comment --comment aaisp6 -j MARK --set-xmark 0x800/0x3f00
-A mwan3_iface_in_henet -i 6in4-henet -m set --match-set mwan3_connected_v6 src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_henet -i 6in4-henet -m mark --mark 0x0/0x3f00 -m comment --comment henet -j MARK --set-xmark 0x900/0x3f00
-A mwan3_iface_in_wanc6 -i eth0.3 -m set --match-set mwan3_connected_v6 src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wanc6 -i eth0.3 -m mark --mark 0x0/0x3f00 -m comment --comment wanc6 -j MARK --set-xmark 0x600/0x3f00
-A mwan3_iface_in_wg6 -i wg -m set --match-set mwan3_connected_v6 src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wg6 -i wg -m mark --mark 0x0/0x3f00 -m comment --comment wg6 -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_iface_in_wgb6 -i wgb -m set --match-set mwan3_connected_v6 src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wgb6 -i wgb -m mark --mark 0x0/0x3f00 -m comment --comment wgb6 -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wanc6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_aaisp6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_henet
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wg6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wgb6
-A mwan3_policy_aaisp_only -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_henet_only -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wan_only -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wan_wanb -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wan_wanb_wanc -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wanb_only -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wanb_wan -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wanc_only -m mark --mark 0x0/0x3f00 -m comment --comment "wanc6 2 2" -j MARK --set-xmark 0x600/0x3f00
-A mwan3_policy_wg_balanced -m mark --mark 0x0/0x3f00 -m statistic --mode random --probability 0.50000000000 -m comment --comment "wgb6 3 6" -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_policy_wg_balanced -m mark --mark 0x0/0x3f00 -m comment --comment "wg6 3 3" -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_policy_wg_only -m mark --mark 0x0/0x3f00 -m comment --comment "wg6 3 3" -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_policy_wgb_only -m mark --mark 0x0/0x3f00 -m comment --comment "wgb6 3 3" -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_rule_https -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb_wanc
-A mwan3_rule_https -m mark ! --mark 0xfc00/0xfc00 -j SET --del-set mwan3_sticky_https src,src
-A mwan3_rule_https -m mark ! --mark 0xfc00/0xfc00 -j SET --add-set mwan3_sticky_https src,src
-A mwan3_rules -p tcp -m multiport --dports 5201 -m mark --mark 0x0/0x3f00 -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_rules -d 2a01:4c8:f000:1::1/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a01:4c8:f000:1::2/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a01:4c8::/29 -p icmp -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a06:24c0::/29 -p icmp -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2001:470:0:64::2/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_henet_only
-A mwan3_rules -p tcp -m multiport --dports 563 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wg_only
-A mwan3_rules -m set --match-set wanb_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_only
-A mwan3_rules -m set --match-set wanc_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -m set --match-set aaisp_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_aaisp_only
-A mwan3_rules -m set --match-set vpn_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wg_balanced
-A mwan3_rules -p tcp -m multiport --dports 443 -m mark --mark 0x0/0x3f00 -j mwan3_rule_https
-A mwan3_rules -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb_wanc

From 2.10.x:

root@linksys-wrt3200acm:~# ip6tables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N postrouting_guest_rule
-N postrouting_lan_rule
-N postrouting_rule
-N postrouting_vpn_rule
-N postrouting_wan_rule
-N postrouting_wireguard_rule
-N prerouting_guest_rule
-N prerouting_lan_rule
-N prerouting_rule
-N prerouting_vpn_rule
-N prerouting_wan_rule
-N prerouting_wireguard_rule
-N zone_guest_postrouting
-N zone_guest_prerouting
-N zone_lan_postrouting
-N zone_lan_prerouting
-N zone_vpn_postrouting
-N zone_vpn_prerouting
-N zone_wan_postrouting
-N zone_wan_prerouting
-N zone_wireguard_postrouting
-N zone_wireguard_prerouting
-A PREROUTING -m comment --comment "!fw3: Custom prerouting rule chain" -j prerouting_rule
-A PREROUTING -i br-lan -m comment --comment "!fw3" -j zone_lan_prerouting
-A PREROUTING -i l2tp-aaisp -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i 6in4-henet -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth0.4 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth1.2 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i pppoe-wanb -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i eth0.3 -m comment --comment "!fw3" -j zone_wan_prerouting
-A PREROUTING -i wlan1-1 -m comment --comment "!fw3" -j zone_guest_prerouting
-A PREROUTING -i tun0 -m comment --comment "!fw3" -j zone_vpn_prerouting
-A PREROUTING -i wg -m comment --comment "!fw3" -j zone_wireguard_prerouting
-A PREROUTING -i wgb -m comment --comment "!fw3" -j zone_wireguard_prerouting
-A POSTROUTING -m comment --comment "!fw3: Custom postrouting rule chain" -j postrouting_rule
-A POSTROUTING -o br-lan -m comment --comment "!fw3" -j zone_lan_postrouting
-A POSTROUTING -o l2tp-aaisp -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o 6in4-henet -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth0.4 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth1.2 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o pppoe-wanb -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o eth0.3 -m comment --comment "!fw3" -j zone_wan_postrouting
-A POSTROUTING -o wlan1-1 -m comment --comment "!fw3" -j zone_guest_postrouting
-A POSTROUTING -o tun0 -m comment --comment "!fw3" -j zone_vpn_postrouting
-A POSTROUTING -o wg -m comment --comment "!fw3" -j zone_wireguard_postrouting
-A POSTROUTING -o wgb -m comment --comment "!fw3" -j zone_wireguard_postrouting
-A zone_guest_postrouting -m comment --comment "!fw3: Custom guest postrouting rule chain" -j postrouting_guest_rule
-A zone_guest_prerouting -m comment --comment "!fw3: Custom guest prerouting rule chain" -j prerouting_guest_rule
-A zone_lan_postrouting -m comment --comment "!fw3: Custom lan postrouting rule chain" -j postrouting_lan_rule
-A zone_lan_prerouting -m comment --comment "!fw3: Custom lan prerouting rule chain" -j prerouting_lan_rule
-A zone_vpn_postrouting -m comment --comment "!fw3: Custom vpn postrouting rule chain" -j postrouting_vpn_rule
-A zone_vpn_prerouting -m comment --comment "!fw3: Custom vpn prerouting rule chain" -j prerouting_vpn_rule
-A zone_wan_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wan_postrouting -m comment --comment "!fw3: Custom wan postrouting rule chain" -j postrouting_wan_rule
-A zone_wan_prerouting -m comment --comment "!fw3: Custom wan prerouting rule chain" -j prerouting_wan_rule
-A zone_wireguard_postrouting -m comment --comment "!fw3" -j MASQUERADE
-A zone_wireguard_postrouting -m comment --comment "!fw3: Custom wireguard postrouting rule chain" -j postrouting_wireguard_rule
-A zone_wireguard_prerouting -m comment --comment "!fw3: Custom wireguard prerouting rule chain" -j prerouting_wireguard_rule

root@linksys-wrt3200acm:~# ip6tables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N mwan3_connected
-N mwan3_hook
-N mwan3_iface_in_aaisp6
-N mwan3_iface_in_henet
-N mwan3_iface_in_wanc6
-N mwan3_iface_in_wg6
-N mwan3_iface_in_wgb6
-N mwan3_ifaces_in
-N mwan3_policy_aaisp_only
-N mwan3_policy_henet_only
-N mwan3_policy_wan_only
-N mwan3_policy_wan_wanb
-N mwan3_policy_wan_wanb_wanc
-N mwan3_policy_wanb_only
-N mwan3_policy_wanb_wan
-N mwan3_policy_wanc_only
-N mwan3_policy_wg_balanced
-N mwan3_policy_wg_only
-N mwan3_policy_wgb_only
-N mwan3_rule_https
-N mwan3_rules
-A PREROUTING -j mwan3_hook
-A FORWARD -o l2tp-aaisp -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i l2tp-aaisp -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o 6in4-henet -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i 6in4-henet -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth0.4 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth1.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth1.2 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o pppoe-wanb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i pppoe-wanb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i eth0.3 -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wan MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o wg -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i wg -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -o wgb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i wgb -p tcp -m tcp --tcp-flags SYN,RST SYN -m comment --comment "!fw3: Zone wireguard MTU fixing" -j TCPMSS --clamp-mss-to-pmtu
-A OUTPUT -j mwan3_hook
-A mwan3_connected -m set --match-set mwan3_connected dst -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 133 -j RETURN
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 134 -j RETURN
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 135 -j RETURN
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 136 -j RETURN
-A mwan3_hook -p ipv6-icmp -m icmp6 --icmpv6-type 137 -j RETURN
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_ifaces_in
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0x3f00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0x3f00 --ctmask 0x3f00
-A mwan3_hook -m mark ! --mark 0x3f00/0x3f00 -j mwan3_connected
-A mwan3_iface_in_aaisp6 -i l2tp-aaisp -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_aaisp6 -i l2tp-aaisp -m mark --mark 0x0/0x3f00 -m comment --comment aaisp6 -j MARK --set-xmark 0x800/0x3f00
-A mwan3_iface_in_henet -i 6in4-henet -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_henet -i 6in4-henet -m mark --mark 0x0/0x3f00 -m comment --comment henet -j MARK --set-xmark 0x900/0x3f00
-A mwan3_iface_in_wanc6 -i eth0.3 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wanc6 -i eth0.3 -m mark --mark 0x0/0x3f00 -m comment --comment wanc6 -j MARK --set-xmark 0x600/0x3f00
-A mwan3_iface_in_wg6 -i wg -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wg6 -i wg -m mark --mark 0x0/0x3f00 -m comment --comment wg6 -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_iface_in_wgb6 -i wgb -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_wgb6 -i wgb -m mark --mark 0x0/0x3f00 -m comment --comment wgb6 -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wanc6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_aaisp6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_henet
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wg6
-A mwan3_ifaces_in -m mark --mark 0x0/0x3f00 -j mwan3_iface_in_wgb6
-A mwan3_policy_aaisp_only -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_henet_only -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wan_only -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wan_wanb -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wan_wanb_wanc -m mark --mark 0x0/0x3f00 -m comment --comment "aaisp6 3 3" -j MARK --set-xmark 0x800/0x3f00
-A mwan3_policy_wanb_only -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wanb_wan -m mark --mark 0x0/0x3f00 -m comment --comment "henet 2 2" -j MARK --set-xmark 0x900/0x3f00
-A mwan3_policy_wanc_only -m mark --mark 0x0/0x3f00 -m comment --comment "wanc6 2 2" -j MARK --set-xmark 0x600/0x3f00
-A mwan3_policy_wg_balanced -m mark --mark 0x0/0x3f00 -m statistic --mode random --probability 0.50000000000 -m comment --comment "wgb6 3 6" -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_policy_wg_balanced -m mark --mark 0x0/0x3f00 -m comment --comment "wg6 3 3" -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_policy_wg_only -m mark --mark 0x0/0x3f00 -m comment --comment "wg6 3 3" -j MARK --set-xmark 0xb00/0x3f00
-A mwan3_policy_wgb_only -m mark --mark 0x0/0x3f00 -m comment --comment "wgb6 3 3" -j MARK --set-xmark 0xd00/0x3f00
-A mwan3_rule_https -m mark --mark 0x0/0x3f00 -j MARK --set-xmark 0x800/0x3f00
-A mwan3_rule_https -m mark --mark 0x800/0x3f00 -m set ! --match-set mwan3_sticky_https src,src -j MARK --set-xmark 0x0/0x3f00
-A mwan3_rule_https -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb_wanc
-A mwan3_rule_https -m mark ! --mark 0xfc00/0xfc00 -j SET --del-set mwan3_sticky_https src,src
-A mwan3_rule_https -m mark ! --mark 0xfc00/0xfc00 -j SET --add-set mwan3_sticky_https src,src
-A mwan3_rules -p tcp -m multiport --dports 5201 -m mark --mark 0x0/0x3f00 -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_rules -d 2a01:4c8:f000:1::1/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a01:4c8:f000:1::2/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a01:4c8::/29 -p icmp -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2a06:24c0::/29 -p icmp -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -d 2001:470:0:64::2/128 -m mark --mark 0x0/0x3f00 -j mwan3_policy_henet_only
-A mwan3_rules -p tcp -m multiport --dports 563 -m mark --mark 0x0/0x3f00 -j mwan3_policy_wg_only
-A mwan3_rules -m set --match-set wanb_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanb_only
-A mwan3_rules -m set --match-set wanc_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wanc_only
-A mwan3_rules -m set --match-set aaisp_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_aaisp_only
-A mwan3_rules -m set --match-set vpn_set dst -m mark --mark 0x0/0x3f00 -j mwan3_policy_wg_balanced
-A mwan3_rules -p tcp -m multiport --dports 443 -m mark --mark 0x0/0x3f00 -j mwan3_rule_https
-A mwan3_rules -m mark --mark 0x0/0x3f00 -j mwan3_policy_wan_wanb_wanc

aaronjg commented 3 years ago

Thanks. I suspect the issue is the change from this:

-A mwan3_hook -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00

to this

-A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00

However, we can't just revert that line. We now need to be able to mark packets to be ignored by mwan3 with the LD_PRELOAD.

So the question is why are the packets being marked 0x3f00 before they enter the mwan3_hook? It must be something related to the NAT setup. Any idea what could be going on?

jamesmacwhite commented 3 years ago

Getting closer!

So the NAT6 script I use is here: https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6, though I can't see why that would do anything related to fwmark. 0x3f00 is the firewall mask value of mwan3 though isn't it.

The IPv6 masquerading is partially broken in fw3, so that's why the NAT6 helper script is needed.

aaronjg commented 3 years ago

The mark is updated to 0x3f00 after:

-A mwan3_hook -j CONNMARK --save-mark --nfmask 0x3f00 --ctmask 0x3f00

-A mwan3_hook -m mark ! --mark 0x3f00/0x3f00 -j mwan3_connected

It seems that NAT6 is saving and restoring this mark with its connection tracking, where NAT4 does not do this. I'm still not clear why though.

jamesmacwhite commented 3 years ago

Interesting find. Nothing on the NAT6 side will have changed though so I assume it has been doing this under 2.8.x as well, it's just the change to -A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00 now exposes the issue, as that's the only difference?

Not really sure of any alternatives. There is a simplified NAT6 configuration, compared to using the full helper library, but I'd assume it will do the same as it's still using kmod-ipt-nat6. There isn't any other IPv6 configuration method I can use, I can't use NETMAP as I don't have a prefix to map for every WAN interface.

aaronjg commented 3 years ago

it's just the change to -A mwan3_hook -m mark --mark 0x0/0x3f00 -j CONNMARK --restore-mark --nfmask 0x3f00 --ctmask 0x3f00 now exposes the issue, as that's the only difference?

I suspect that is the case. Can you confirm by removing line 297 here: https://github.com/openwrt/packages/blob/master/net/mwan3/files/lib/mwan3/mwan3.sh#L296-L298 ?

That fix will be fine on 19.07.x, but this will render useless the LD_PRELOAD mwan3track fix with the new kernel in 20.x, so not a great long term solution.

Maybe there is a bug upstream with kmod-ipt-nat6?

jamesmacwhite commented 3 years ago

I'll see if I can test that theory out soon.

My next question was going to be I wonder if NAT6 behaves this way on the 5.4 kernel. I'm loathed to use a snapshot build for a production setting, but perhaps it's not broken like this on snapshot. It is not the first time something in the kernel has caused issues with mwan3, even though it's not directly related, the fwmark issue was a fun one.

There's one other person on the forums recently who was using 2.10.x with NAT6, they had a different IPv6 problem, so I'm not quite sure why I get the 50% loss issue, I removed all my rules except for the default, it does seem to be specific to NAT6 though.

wackejohn commented 3 years ago

Hi, all I'm using mwan3 2.10.5-1 with IPv6 NAT on latest snapshot (kernel version 5.10) and my firewall package was patched to support IPv6 NAT directlly with the patch below:

diff -uNr firewall_orig/defaults.c firewall/defaults.c
--- firewall_orig/defaults.c    2020-07-25 08:54:12.000000000 +0800
+++ firewall/defaults.c 2020-08-08 20:26:45.678198422 +0800
@@ -29,8 +29,8 @@
    C(ANY, FILTER, CUSTOM_CHAINS, "forwarding_rule"),
    C(ANY, FILTER, SYN_FLOOD,     "syn_flood"),

-   C(V4,  NAT,    CUSTOM_CHAINS, "prerouting_rule"),
-   C(V4,  NAT,    CUSTOM_CHAINS, "postrouting_rule"),
+   C(ANY, NAT,    CUSTOM_CHAINS, "prerouting_rule"),
+   C(ANY, NAT,    CUSTOM_CHAINS, "postrouting_rule"),

    { }
 };
@@ -251,7 +251,8 @@
            }
        }

-       if (defs->flow_offloading)
+       // There seemd to be issues with offloading for nat6. Only use it on IPv4 for now.
+       if (defs->flow_offloading && handle->family == FW3_FAMILY_V4)
        {
            r = fw3_ipt_rule_new(handle);
            fw3_ipt_rule_comment(r, "Traffic offloading");
diff -uNr firewall_orig/options.h firewall/options.h
--- firewall_orig/options.h 2020-07-25 08:54:12.000000000 +0800
+++ firewall/options.h  2020-08-08 17:56:11.852760994 +0800
@@ -263,6 +263,8 @@

 struct fw3_mark
 {
+   struct list_head list;
+
    bool set;
    bool invert;
    uint32_t mark;
@@ -340,6 +342,9 @@
    bool masq_allow_invalid;
    struct list_head masq_src;
    struct list_head masq_dest;
+    
+   bool masq6;
+

    bool mtu_fix;

@@ -420,14 +425,14 @@

    struct list_head proto;

-   struct fw3_address ip_src;
+   struct list_head ip_src;
    struct list_head mac_src;
    struct fw3_port port_src;

-   struct fw3_address ip_dest;
+   struct list_head ip_dest;
    struct fw3_port port_dest;

-   struct fw3_address ip_redir;
+   struct list_head ip_redir;
    struct fw3_port port_redir;

    struct fw3_limit limit;
@@ -462,13 +467,13 @@

    struct list_head proto;

-   struct fw3_address ip_src;
+   struct list_head ip_src;
    struct fw3_port port_src;

-   struct fw3_address ip_dest;
+   struct list_head ip_dest;
    struct fw3_port port_dest;

-   struct fw3_address ip_snat;
+   struct list_head ip_snat;
    struct fw3_port port_snat;

    struct fw3_limit limit;
diff -uNr firewall_orig/redirects.c firewall/redirects.c
--- firewall_orig/redirects.c   2020-07-25 08:54:12.000000000 +0800
+++ firewall/redirects.c    2020-08-08 21:05:46.106316767 +0800
@@ -33,14 +33,14 @@

    FW3_LIST("proto",              protocol,  redirect,     proto),

-   FW3_OPT("src_ip",              network,   redirect,     ip_src),
+   FW3_LIST("src_ip",             network,   redirect,     ip_src),
    FW3_LIST("src_mac",            mac,       redirect,     mac_src),
    FW3_OPT("src_port",            port,      redirect,     port_src),

-   FW3_OPT("src_dip",             network,   redirect,     ip_dest),
+   FW3_LIST("src_dip",            network,   redirect,     ip_dest),
    FW3_OPT("src_dport",           port,      redirect,     port_dest),

-   FW3_OPT("dest_ip",             network,   redirect,     ip_redir),
+   FW3_LIST("dest_ip",            network,   redirect,     ip_redir),
    FW3_OPT("dest_port",           port,      redirect,     port_redir),

    FW3_OPT("extra",               string,    redirect,     extra),
@@ -101,19 +101,19 @@
        return false;
    }

-   if (r->ip_src.family && r->ip_src.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_src, r->family))
    {
        warn_elem(e, "uses source ip with different family");
        return false;
    }

-   if (r->ip_dest.family && r->ip_dest.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_dest, r->family))
    {
        warn_elem(e, "uses destination ip with different family");
        return false;
    }

-   if (r->ip_redir.family && r->ip_redir.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_redir, r->family))
    {
        warn_elem(e, "uses redirect ip with different family");
        return false;
@@ -123,14 +123,24 @@
 }

 static bool
-compare_addr(struct fw3_address *a, struct fw3_address *b)
-{
-   if (a->family != FW3_FAMILY_V4 || b->family != FW3_FAMILY_V4)
-       return false;
-
-   return ((a->address.v4.s_addr & a->mask.v4.s_addr) ==
-           (b->address.v4.s_addr & a->mask.v4.s_addr));
-}
+addr_matches_prefix(struct fw3_address *addr, struct fw3_address *pfx)
+ {
+   if (addr->family != pfx->family)
+       return false;
+ 
+   return (pfx->family == FW3_FAMILY_V4 &&
+           (pfx->address.v4.s_addr & pfx->mask.v4.s_addr) ==
+           (addr->address.v4.s_addr & pfx->mask.v4.s_addr)) ||
+          (pfx->family == FW3_FAMILY_V6 &&
+           (pfx->address.v6.s6_addr32[0] & pfx->mask.v6.s6_addr32[0]) ==
+           (addr->address.v6.s6_addr32[0] & pfx->mask.v6.s6_addr32[0]) &&
+           (pfx->address.v6.s6_addr32[1] & pfx->mask.v6.s6_addr32[1]) ==
+           (addr->address.v6.s6_addr32[1] & pfx->mask.v6.s6_addr32[1]) &&
+           (pfx->address.v6.s6_addr32[2] & pfx->mask.v6.s6_addr32[2]) ==
+           (addr->address.v6.s6_addr32[2] & pfx->mask.v6.s6_addr32[2]) &&
+           (pfx->address.v6.s6_addr32[3] & pfx->mask.v6.s6_addr32[3]) ==
+           (addr->address.v6.s6_addr32[3] & pfx->mask.v6.s6_addr32[3]));
+ }

 static bool
 resolve_dest(struct uci_element *e, struct fw3_redirect *redir,
@@ -139,8 +149,9 @@
    struct fw3_zone *zone;
    struct fw3_address *addr;
    struct list_head *addrs;
+   struct fw3_address *ip_redir = list_first_entry(&redir->ip_redir, struct fw3_address, list);

-   if (!redir->ip_redir.set)
+   if (!ip_redir->set)
        return false;

    list_for_each_entry(zone, &state->zones, list)
@@ -152,7 +163,7 @@

        list_for_each_entry(addr, addrs, list)
        {
-           if (!compare_addr(addr, &redir->ip_redir))
+           if (!addr_matches_prefix(ip_redir, addr))
                continue;

            snprintf(redir->dest.name, sizeof(redir->dest.name), "%s", zone->name);
@@ -175,10 +186,12 @@
 check_local(struct uci_element *e, struct fw3_redirect *redir,
             struct fw3_state *state)
 {
+   struct fw3_address *ip_redir = list_first_entry(&redir->ip_redir, struct fw3_address, list);
+
    if (redir->target != FW3_FLAG_DNAT)
        return false;

-   if (!redir->ip_redir.set)
+   if (!ip_redir->set)
        redir->local = true;

    return redir->local;
@@ -224,6 +237,9 @@
    redir->helper.ptr = helper;

    set(redir->_src->flags, FW3_FAMILY_V4, FW3_FLAG_HELPER);
+
+   set(redir->_src->flags, FW3_FAMILY_V6, FW3_FLAG_HELPER);
+
 }

 static bool
@@ -299,7 +315,11 @@
            warn_section("redirect", redir, e, "must not use a negated helper match");
        else
        {
-           set(redir->_src->flags, FW3_FAMILY_V4, redir->target);
+           if (fw3_is_family(redir, FW3_FAMILY_V6))
+               set(redir->_src->flags, FW3_FAMILY_V6, redir->target);
+           if (fw3_is_family(redir, FW3_FAMILY_V4))
+               set(redir->_src->flags, FW3_FAMILY_V4, redir->target);
+
            valid = true;

            if (!check_local(e, redir, state) && !redir->dest.set &&
@@ -310,15 +330,29 @@
                        redir->dest.name);
            }

-           if (redir->reflection && redir->_dest && redir->_src->masq)
+           if (redir->reflection && redir->_dest)
            {
-               set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_ACCEPT);
-               set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_DNAT);
-               set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_SNAT);
+               if (redir->_src->masq && fw3_is_family(redir->_dest, FW3_FAMILY_V4)) {
+                   set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_ACCEPT);
+                   set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_DNAT);
+                   set(redir->_dest->flags, FW3_FAMILY_V4, FW3_FLAG_SNAT);
+               }
+
+               if (redir->_src->masq6 && fw3_is_family(redir->_dest, FW3_FAMILY_V6)) {
+                   set(redir->_dest->flags, FW3_FAMILY_V6, FW3_FLAG_ACCEPT);
+                   set(redir->_dest->flags, FW3_FAMILY_V6, FW3_FLAG_DNAT);
+                   set(redir->_dest->flags, FW3_FAMILY_V6, FW3_FLAG_SNAT);
+               }
            }

            if (redir->helper.ptr)
+           {
                set(redir->_src->flags, FW3_FAMILY_V4, FW3_FLAG_HELPER);
+
+               set(redir->_src->flags, FW3_FAMILY_V6, FW3_FLAG_HELPER);
+
+           }
+
        }
    }
    else
@@ -328,7 +362,7 @@
                    "must not have destination '*' for SNAT target");
        else if (!redir->_dest)
            warn_section("redirect", redir, e, "has no destination specified");
-       else if (!redir->ip_dest.set)
+       else if (list_empty(&redir->ip_dest))
            warn_section("redirect", redir, e, "has no src_dip option specified");
        else if (!list_empty(&redir->mac_src))
            warn_section("redirect", redir, e, "must not use 'src_mac' option for SNAT target");
@@ -337,6 +371,9 @@
        else
        {
            set(redir->_dest->flags, FW3_FAMILY_V4, redir->target);
+
+           set(redir->_dest->flags, FW3_FAMILY_V6, redir->target);
+
            valid = true;
        }
    }
@@ -366,7 +403,10 @@
        return NULL;

    INIT_LIST_HEAD(&redir->proto);
+   INIT_LIST_HEAD(&redir->ip_src);
    INIT_LIST_HEAD(&redir->mac_src);
+   INIT_LIST_HEAD(&redir->ip_dest);
+   INIT_LIST_HEAD(&redir->ip_redir);
    INIT_LIST_HEAD(&redir->reflection_zones);

    redir->enabled = true;
@@ -477,14 +517,17 @@
 set_snat_dnat(struct fw3_ipt_rule *r, enum fw3_flag target,
               struct fw3_address *addr, struct fw3_port *port)
 {
-   char buf[sizeof("255.255.255.255:65535-65535")] = {};
+   char buf[sizeof("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff:65535-65535")] = {};
    char ip[INET_ADDRSTRLEN], *p = buf;
    size_t rem = sizeof(buf);
    int len;

    if (addr && addr->set)
    {
-       inet_ntop(AF_INET, &addr->address.v4, ip, sizeof(ip));
+       if (addr->family == FW3_FAMILY_V6) 
+           inet_ntop(AF_INET6, &addr->address.v6, ip, sizeof(ip));
+       else
+           inet_ntop(AF_INET, &addr->address.v4, ip, sizeof(ip));

        len = snprintf(p, rem, "%s", ip);

@@ -516,17 +559,6 @@
 }

 static void
-set_target_nat(struct fw3_ipt_rule *r, struct fw3_redirect *redir)
-{
-   if (redir->local)
-       set_redirect(r, &redir->port_redir);
-   else if (redir->target == FW3_FLAG_DNAT)
-       set_snat_dnat(r, redir->target, &redir->ip_redir, &redir->port_redir);
-   else
-       set_snat_dnat(r, redir->target, &redir->ip_dest, &redir->port_dest);
-}
-
-static void
 set_comment(struct fw3_ipt_rule *r, const char *name, int num, const char *suffix)
 {
    if (name)
@@ -551,32 +583,41 @@
                struct fw3_protocol *proto, struct fw3_mac *mac)
 {
    struct fw3_ipt_rule *r;
-   struct fw3_address *src, *dst;
-   struct fw3_port *spt, *dpt;
+   struct fw3_address *src, *dst, *rdr;
+   struct fw3_port *spt, *dpt, *rpt;

    switch (h->table)
    {
    case FW3_TABLE_NAT:
-       src = &redir->ip_src;
-       dst = &redir->ip_dest;
-       spt = &redir->port_src;
-       dpt = &redir->port_dest;
-
-       if (redir->target == FW3_FLAG_SNAT)
+       src = fw3_first_family_addr(&redir->ip_src, h->family);
+       dst = fw3_first_family_addr(&redir->ip_dest, h->family);
+       rdr = fw3_first_family_addr(&redir->ip_redir, h->family);
+       if ((src == NULL && !list_empty(&redir->ip_src)) ||
+           (dst == NULL && !list_empty(&redir->ip_dest)) ||
+           (rdr == NULL && !list_empty(&redir->ip_redir)))
        {
-           dst = &redir->ip_redir;
-           dpt = &redir->port_redir;
+           info("     ! Skipping due to different family of ip address");
+           return;
        }

-       r = fw3_ipt_rule_create(h, proto, NULL, NULL, src, dst);
-       fw3_ipt_rule_sport_dport(r, spt, dpt);
+       spt = &redir->port_src;
+       dpt = &redir->port_dest;
+       rpt = &redir->port_redir;
+
+       r = fw3_ipt_rule_create(h, proto, NULL, NULL, src, redir->target == FW3_FLAG_SNAT ? rdr : dst);
+       fw3_ipt_rule_sport_dport(r, spt, redir->target == FW3_FLAG_SNAT ? rpt : dpt);
        fw3_ipt_rule_mac(r, mac);
        fw3_ipt_rule_ipset(r, &redir->ipset);
        fw3_ipt_rule_helper(r, &redir->helper);
        fw3_ipt_rule_limit(r, &redir->limit);
        fw3_ipt_rule_time(r, &redir->time);
        fw3_ipt_rule_mark(r, &redir->mark);
-       set_target_nat(r, redir);
+       if (redir->local)
+           set_redirect(r, rpt);
+       else if (redir->target == FW3_FLAG_DNAT)
+           set_snat_dnat(r, redir->target, rdr, rpt);
+       else
+           set_snat_dnat(r, redir->target, dst, dpt);
        fw3_ipt_rule_extra(r, redir->extra);
        set_comment(r, redir->name, num, NULL);
        append_chain_nat(r, redir);
@@ -585,6 +626,15 @@
    case FW3_TABLE_RAW:
        if (redir->target == FW3_FLAG_DNAT && redir->helper.ptr)
        {
+           src = fw3_first_family_addr(&redir->ip_src, h->family);
+           rdr = fw3_first_family_addr(&redir->ip_redir, h->family);
+           if ((src == NULL && !list_empty(&redir->ip_src)) ||
+               (rdr == NULL && !list_empty(&redir->ip_redir)))
+           {
+               info("     ! Skipping due to different family of ip address");
+               return;
+           }
+
            if (!fw3_cthelper_check_proto(redir->helper.ptr, proto))
            {
                info("     ! Skipping protocol %s since helper '%s' does not support it",
@@ -596,7 +646,7 @@
                info("     - Auto-selected conntrack helper '%s' based on proto/port",
                     redir->helper.ptr->name);

-           r = fw3_ipt_rule_create(h, proto, NULL, NULL, &redir->ip_src, &redir->ip_redir);
+           r = fw3_ipt_rule_create(h, proto, NULL, NULL, src, rdr);
            fw3_ipt_rule_sport_dport(r, &redir->port_src, &redir->port_redir);
            fw3_ipt_rule_mac(r, mac);
            fw3_ipt_rule_ipset(r, &redir->ipset);
@@ -624,19 +674,26 @@
                  struct fw3_address *ia, struct fw3_address *ea, struct fw3_device *rz)
 {
    struct fw3_ipt_rule *r;
+   struct fw3_address *rdr;

    switch (h->table)
    {
    case FW3_TABLE_NAT:
+       rdr = fw3_first_family_addr(&redir->ip_redir, h->family);
+       if (rdr == NULL && !list_empty(&redir->ip_redir))
+       {
+           info("     ! Skipping reflection due to different family of dest_ip");
+           return;
+       }
        r = fw3_ipt_rule_create(h, proto, NULL, NULL, ia, ea);
        fw3_ipt_rule_sport_dport(r, NULL, &redir->port_dest);
        fw3_ipt_rule_limit(r, &redir->limit);
        fw3_ipt_rule_time(r, &redir->time);
        set_comment(r, redir->name, num, "reflection");
-       set_snat_dnat(r, FW3_FLAG_DNAT, &redir->ip_redir, &redir->port_redir);
+       set_snat_dnat(r, FW3_FLAG_DNAT, rdr, &redir->port_redir);
        fw3_ipt_rule_replace(r, "zone_%s_prerouting", rz->name);

-       r = fw3_ipt_rule_create(h, proto, NULL, NULL, ia, &redir->ip_redir);
+       r = fw3_ipt_rule_create(h, proto, NULL, NULL, ia, rdr);
        fw3_ipt_rule_sport_dport(r, NULL, &redir->port_redir);
        fw3_ipt_rule_limit(r, &redir->limit);
        fw3_ipt_rule_time(r, &redir->time);
@@ -660,6 +717,9 @@
    struct fw3_mac *mac;
    struct fw3_device *reflection_zone;
    struct fw3_zone *zone;
+    
+   if (!fw3_is_family(redir, handle->family))
+       return;

    if (redir->name)
        info("   * Redirect '%s'", redir->name);
@@ -673,15 +733,11 @@
        return;
    }

-   if (!fw3_is_family(&redir->ip_src, handle->family) ||
-       !fw3_is_family(&redir->ip_dest, handle->family) ||
-       !fw3_is_family(&redir->ip_redir, handle->family))
-   {
-       if (!redir->ip_src.resolved ||
-           !redir->ip_dest.resolved ||
-           !redir->ip_redir.resolved)
-           info("     ! Skipping due to different family of ip address");
-
+   if (!fw3_check_family_addr(&redir->ip_src, handle->family) ||
+       !fw3_check_family_addr(&redir->ip_dest, handle->family) ||
+       !fw3_check_family_addr(&redir->ip_redir, handle->family))
+   {
+       info("     ! Skipping due to different family of ip address");
        return;
    }

@@ -709,13 +765,13 @@
        print_redirect(handle, state, redir, num, proto, mac);

    /* reflection rules */
-   if (redir->target != FW3_FLAG_DNAT || !redir->reflection || redir->local)
+   if (redir->target != FW3_FLAG_DNAT || !redir->reflection || redir->local || handle->family != FW3_FAMILY_V4)
        return;

    if (!redir->_dest || !redir->_src->masq)
        return;

-   ext_addrs = fw3_resolve_zone_addresses(redir->_src, &redir->ip_dest);
+   ext_addrs = fw3_resolve_zone_addresses(redir->_src, fw3_first_family_addr(&redir->ip_dest, handle->family));
    if (!ext_addrs)
        return;

@@ -758,8 +814,15 @@
                    else
                        ref_addr = *ext_addr;

-                   ref_addr.mask.v4.s_addr = 0xFFFFFFFF;
-                   ext_addr->mask.v4.s_addr = 0xFFFFFFFF;
+                   if (ref_addr.family == FW3_FAMILY_V6)
+                       memset(ref_addr.mask.v6.s6_addr, 0xFF, 16);
+                   else
+                       ref_addr.mask.v4.s_addr = 0xFFFFFFFF;
+
+                   if (ext_addr->family == FW3_FAMILY_V6)
+                       memset(ext_addr->mask.v6.s6_addr, 0xFF, 16);
+                   else 
+                       ext_addr->mask.v4.s_addr = 0xFFFFFFFF;

                    print_reflection(handle, state, redir, num, proto,
                                     &ref_addr, int_addr, ext_addr, reflection_zone);
@@ -778,7 +841,7 @@
 {
    int num = 0;
    struct fw3_redirect *redir;
-
+    
    if (handle->family == FW3_FAMILY_V6)
        return;

diff -uNr firewall_orig/redirects.h firewall/redirects.h
--- firewall_orig/redirects.h   2020-07-25 08:54:12.000000000 +0800
+++ firewall/redirects.h    2020-08-08 15:18:00.415839357 +0800
@@ -23,7 +23,7 @@
 #include "zones.h"
 #include "ipsets.h"
 #include "helpers.h"
-#include "ubus.h"
+#include "utils.h"
 #include "iptables.h"

 extern const struct fw3_option fw3_redirect_opts[];
diff -uNr firewall_orig/snats.c firewall/snats.c
--- firewall_orig/snats.c   2020-07-25 08:54:12.000000000 +0800
+++ firewall/snats.c    2020-08-08 21:13:59.137916922 +0800
@@ -32,13 +32,13 @@

    FW3_LIST("proto",              protocol,  snat,     proto),

-   FW3_OPT("src_ip",              network,   snat,     ip_src),
+   FW3_LIST("src_ip",             network,   snat,     ip_src),
    FW3_OPT("src_port",            port,      snat,     port_src),

-   FW3_OPT("snat_ip",             network,   snat,     ip_snat),
+   FW3_LIST("snat_ip",            network,   snat,     ip_snat),
    FW3_OPT("snat_port",           port,      snat,     port_snat),

-   FW3_OPT("dest_ip",             network,   snat,     ip_dest),
+   FW3_LIST("dest_ip",            network,   snat,     ip_dest),
    FW3_OPT("dest_port",           port,      snat,     port_dest),

    FW3_OPT("extra",               string,    snat,     extra),
@@ -83,19 +83,19 @@
        return false;
    }

-   if (r->ip_src.family && r->ip_src.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_src, r->family))
    {
        warn_section("nat", r, e, "uses source ip with different family");
        return false;
    }

-   if (r->ip_dest.family && r->ip_dest.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_dest, r->family))
    {
        warn_section("nat", r, e, "uses destination ip with different family");
        return false;
    }

-   if (r->ip_snat.family && r->ip_snat.family != r->family)
+   if (!fw3_check_family_addr(&r->ip_snat, r->family))
    {
        warn_section("nat", r, e, "uses snat ip with different family");
        return false;
@@ -112,6 +112,9 @@

    if (snat) {
        INIT_LIST_HEAD(&snat->proto);
+       INIT_LIST_HEAD(&snat->ip_src);
+       INIT_LIST_HEAD(&snat->ip_dest);
+       INIT_LIST_HEAD(&snat->ip_snat);
        list_add_tail(&snat->list, &state->snats);
        snat->enabled = true;
    }
@@ -164,12 +167,12 @@
    }

    if (snat->target == FW3_FLAG_SNAT &&
-           !snat->ip_snat.set && !snat->port_snat.set)
+           list_empty(&snat->ip_snat) && !snat->port_snat.set)
    {
        warn_section("nat", snat, e, "needs either 'snat_ip' or 'snat_port' for SNAT");
        return false;
    }
-   else if (snat->target != FW3_FLAG_SNAT && snat->ip_snat.set)
+   else if (snat->target != FW3_FLAG_SNAT && !list_empty(&snat->ip_snat))
    {
        warn_section("nat", snat, e, "must not use 'snat_ip' for non-SNAT");
        return false;
@@ -186,8 +189,13 @@
        fw3_parse_protocol(&snat->proto, "all", true);
    }

-   if (snat->_src)
-       set(snat->_src->flags, FW3_FAMILY_V4, FW3_FLAG_SNAT);
+   if (snat->_src) {
+
+       if (fw3_is_family(snat->_src, FW3_FAMILY_V6)) 
+           set(snat->_src->flags, FW3_FAMILY_V6, FW3_FLAG_SNAT);
+       if (fw3_is_family(snat->_src, FW3_FAMILY_V4))
+           set(snat->_src->flags, FW3_FAMILY_V4, FW3_FLAG_SNAT);
+   }

    return true;
 }
@@ -262,19 +270,23 @@
 }

 static void
-set_target(struct fw3_ipt_rule *r, struct fw3_snat *snat,
+set_target(struct fw3_ipt_rule *r, struct fw3_snat *snat, struct fw3_address *snat_addr,
            struct fw3_protocol *proto)
 {
-   char buf[sizeof("255.255.255.255:65535-65535")] = {};
+   char buf[sizeof("ffff:ffff:ffff:ffff:ffff:ffff:ffff:ffff:65535-65535")] = {};
    char ip[INET_ADDRSTRLEN], portcntbuf[6], *p = buf;
    size_t rem = sizeof(buf);
    int len;

    if (snat->target == FW3_FLAG_SNAT)
    {
-       if (snat->ip_snat.set)
+       if (snat_addr && snat_addr->set)
        {
-           inet_ntop(AF_INET, &snat->ip_snat.address.v4, ip, sizeof(ip));
+           
+           if (snat_addr->family == FW3_FAMILY_V6)
+               inet_ntop(AF_INET6, &snat_addr->address.v6, ip, sizeof(ip));
+           else
+               inet_ntop(AF_INET, &snat_addr->address.v4, ip, sizeof(ip));

            len = snprintf(p, rem, "%s", ip);

@@ -331,14 +343,23 @@
            struct fw3_snat *snat, int num, struct fw3_protocol *proto)
 {
    struct fw3_ipt_rule *r;
-   struct fw3_address *src, *dst;
+   struct fw3_address *src, *dst, *snt;
    struct fw3_port *spt, *dpt;

    switch (h->table)
    {
    case FW3_TABLE_NAT:
-       src = &snat->ip_src;
-       dst = &snat->ip_dest;
+       src = fw3_first_family_addr(&snat->ip_src, h->family);
+       dst = fw3_first_family_addr(&snat->ip_dest, h->family);
+       snt = fw3_first_family_addr(&snat->ip_snat, h->family);
+       if ((src == NULL && !list_empty(&snat->ip_src)) ||
+           (dst == NULL && !list_empty(&snat->ip_dest)) ||
+           (snt == NULL && !list_empty(&snat->ip_snat)))
+       {
+           info("     ! Skipping due to different family of ip address");
+           return;
+       }
+
        spt = &snat->port_src;
        dpt = &snat->port_dest;

@@ -349,7 +370,7 @@
        fw3_ipt_rule_limit(r, &snat->limit);
        fw3_ipt_rule_time(r, &snat->time);
        fw3_ipt_rule_mark(r, &snat->mark);
-       set_target(r, snat, proto);
+       set_target(r, snat, snt, proto);
        fw3_ipt_rule_extra(r, snat->extra);
        set_comment(r, snat->name, num);
        append_chain(r, snat);
@@ -366,6 +387,9 @@
 {
    struct fw3_protocol *proto;

+   if (!fw3_is_family(snat, handle->family))
+       return;
+
    if (snat->name)
        info("   * NAT '%s'", snat->name);
    else
@@ -377,15 +401,11 @@
        return;
    }

-   if (!fw3_is_family(&snat->ip_src, handle->family) ||
-       !fw3_is_family(&snat->ip_dest, handle->family) ||
-       !fw3_is_family(&snat->ip_snat, handle->family))
-   {
-       if (!snat->ip_src.resolved ||
-           !snat->ip_dest.resolved ||
-           !snat->ip_snat.resolved)
-           info("     ! Skipping due to different family of ip address");
-
+   if (!fw3_check_family_addr(&snat->ip_src, handle->family) ||
+       !fw3_check_family_addr(&snat->ip_dest, handle->family) ||
+       !fw3_check_family_addr(&snat->ip_snat, handle->family))
+   {
+       info("     ! Skipping due to different family of ip address");
        return;
    }

@@ -418,8 +438,6 @@
    int num = 0;
    struct fw3_snat *snat;

-   if (handle->family == FW3_FAMILY_V6)
-       return;

    if (handle->table != FW3_TABLE_NAT)
        return;
diff -uNr firewall_orig/ubus.c firewall/ubus.c
--- firewall_orig/ubus.c    2020-07-25 08:54:12.000000000 +0800
+++ firewall/ubus.c 2020-08-08 15:39:39.022044603 +0800
@@ -133,6 +133,42 @@
    return n;
 }

+static int
+parse_prefix_assignments(struct list_head *head, enum fw3_family family,
+              struct blob_attr *list)
+{
+   struct blob_attr *pfx, *cur;
+   struct fw3_address *addr;
+   int rem, pfxlen, n = 0;
+
+   if (!list)
+       return 0;
+
+   rem = blobmsg_data_len(list);
+
+   __blob_for_each_attr(pfx, blobmsg_data(list), rem)
+   {
+       pfxlen = blobmsg_data_len(pfx);
+
+       __blob_for_each_attr(cur, blobmsg_data(pfx), pfxlen)
+       {
+           if (!strcmp(blobmsg_name(cur), "local-address"))
+           {
+               addr = parse_subnet(family, blobmsg_data(cur), blobmsg_data_len(cur));
+
+               if (addr)
+               {
+                   list_add_tail(&addr->list, head);
+                   n++;
+               }
+           }
+       }
+   }
+
+   return n;
+}
+
+
 struct fw3_device *
 fw3_ubus_device(const char *net)
 {
@@ -218,7 +254,7 @@

        n += parse_subnets(list, FW3_FAMILY_V4, tb[ADDR_IPV4]);
        n += parse_subnets(list, FW3_FAMILY_V6, tb[ADDR_IPV6]);
-       n += parse_subnets(list, FW3_FAMILY_V6, tb[ADDR_IPV6_PREFIX]);
+       n += parse_prefix_assignments(list, FW3_FAMILY_V6, tb[ADDR_IPV6_PREFIX]);
    }

    return n;
diff -uNr firewall_orig/utils.c firewall/utils.c
--- firewall_orig/utils.c   2020-07-25 08:54:12.000000000 +0800
+++ firewall/utils.c    2020-08-08 15:43:47.273859469 +0800
@@ -510,6 +510,11 @@
    ptr.option = "masq";
    ptr.value  = z->masq ? "1" : "0";
    uci_set(ctx, &ptr);
+    
+   ptr.o      = NULL;
+   ptr.option = "masq6";
+   ptr.value  = z->masq6 ? "1" : "0";
+   uci_set(ctx, &ptr);

    ptr.o      = NULL;
    ptr.option = "mtu_fix";
@@ -1046,3 +1051,37 @@

    return false;
 }
+
+struct fw3_address *
+fw3_first_family_addr(struct list_head *addrs, enum fw3_family family)
+{
+   struct fw3_address *addr;
+
+   if (family == FW3_FAMILY_ANY)
+       return NULL;
+
+   list_for_each_entry(addr, addrs, list)
+   {
+       if (addr->family == family)
+           return addr;
+   }
+
+   return NULL;
+}
+
+bool
+fw3_check_family_addr(struct list_head *addrs, enum fw3_family family)
+{
+   struct fw3_address *addr;
+
+   if (family == FW3_FAMILY_ANY || list_empty(addrs))
+       return true;
+
+   list_for_each_entry(addr, addrs, list)
+   {
+       if (addr->family == family)
+           return true;
+   }
+
+   return false;
+}
diff -uNr firewall_orig/utils.h firewall/utils.h
--- firewall_orig/utils.h   2020-07-25 08:54:12.000000000 +0800
+++ firewall/utils.h    2020-08-08 15:47:01.854530027 +0800
@@ -44,6 +44,7 @@

 extern bool fw3_pr_debug;

+enum fw3_family;
 struct fw3_address;

 void warn_elem(struct uci_element *e, const char *format, ...)
@@ -134,4 +135,9 @@
 bool fw3_check_loopback_dev(const char *name);

 bool fw3_check_loopback_addr(struct fw3_address *addr);
+
+struct fw3_address * fw3_first_family_addr(struct list_head *addrs, enum fw3_family family);
+
+bool fw3_check_family_addr(struct list_head *addrs, enum fw3_family family);
+
 #endif
diff -uNr firewall_orig/zones.c firewall/zones.c
--- firewall_orig/zones.c   2020-07-25 08:54:12.000000000 +0800
+++ firewall/zones.c    2020-08-08 20:41:00.371634182 +0800
@@ -37,8 +37,8 @@
    C(ANY, FILTER, REJECT,        "zone_?_dest_REJECT"),
    C(ANY, FILTER, DROP,          "zone_?_dest_DROP"),

-   C(V4,  NAT,    SNAT,          "zone_?_postrouting"),
-   C(V4,  NAT,    DNAT,          "zone_?_prerouting"),
+   C(ANY, NAT,    SNAT,          "zone_?_postrouting"),
+   C(ANY, NAT,    DNAT,          "zone_?_prerouting"),

    C(ANY, RAW,    HELPER,        "zone_?_helper"),
    C(ANY, RAW,    NOTRACK,       "zone_?_notrack"),
@@ -47,8 +47,8 @@
    C(ANY, FILTER, CUSTOM_CHAINS, "output_?_rule"),
    C(ANY, FILTER, CUSTOM_CHAINS, "forwarding_?_rule"),

-   C(V4,  NAT,    CUSTOM_CHAINS, "prerouting_?_rule"),
-   C(V4,  NAT,    CUSTOM_CHAINS, "postrouting_?_rule"),
+   C(ANY, NAT,    CUSTOM_CHAINS, "prerouting_?_rule"),
+   C(ANY, NAT,    CUSTOM_CHAINS, "postrouting_?_rule"),

    { }
 };
@@ -77,6 +77,8 @@
    FW3_LIST("masq_src",           network,  zone,     masq_src),
    FW3_LIST("masq_dest",          network,  zone,     masq_dest),

+   FW3_OPT("masq6",               bool,     zone,     masq6),
+
    FW3_OPT("extra",               string,   zone,     extra_src),
    FW3_OPT("extra_src",           string,   zone,     extra_src),
    FW3_OPT("extra_dest",          string,   zone,     extra_dest),
@@ -306,10 +308,18 @@
            fw3_setbit(zone->flags[0], FW3_FLAG_SNAT);
        }

+       if (zone->masq6) 
+       {
+           fw3_setbit(zone->flags[1], FW3_FLAG_SNAT);
+       }
+
        if (zone->custom_chains)
        {
            fw3_setbit(zone->flags[0], FW3_FLAG_SNAT);
            fw3_setbit(zone->flags[0], FW3_FLAG_DNAT);
+
+           fw3_setbit(zone->flags[1], FW3_FLAG_SNAT);
+           fw3_setbit(zone->flags[1], FW3_FLAG_DNAT);
        }

        resolve_cthelpers(state, e, zone);
@@ -705,7 +715,7 @@
        break;

    case FW3_TABLE_NAT:
-       if (zone->masq && handle->family == FW3_FAMILY_V4)
+       if ((zone->masq && handle->family == FW3_FAMILY_V4) || (zone->masq6 && handle->family == FW3_FAMILY_V6))
        {
            /* for any negated masq_src ip, emit -s addr -j RETURN rules */
            for (msrc = NULL;

I just picked up the patch from mailing list: https://patchwork.ozlabs.org/project/openwrt/patch/1588850061-9861-1-git-send-email-alin.nastac@gmail.com/ https://patchwork.ozlabs.org/project/openwrt/patch/1431126594-6375-2-git-send-email-larsg@systemli.org/ https://patchwork.ozlabs.org/project/openwrt/patch/1431126594-6375-3-git-send-email-larsg@systemli.org/

And my clients after router didn't have this issue:

Wacke@HOME-Server:~> ping6 ipv6.google.com
PING ipv6.google.com(lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e)) 56 data bytes
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=1 ttl=118 time=136 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=2 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=3 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=4 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=5 ttl=118 time=134 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=6 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=7 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=8 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=9 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=10 ttl=118 time=136 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=11 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=12 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=13 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=14 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=15 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=16 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=17 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=18 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=19 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=20 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=21 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=22 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=23 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=24 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=25 ttl=118 time=136 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=26 ttl=118 time=135 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=27 ttl=118 time=136 ms
64 bytes from lax17s44-in-x0e.1e100.net (2607:f8b0:4007:80f::200e): icmp_seq=28 ttl=118 time=136 ms
^C
--- ipv6.google.com ping statistics ---
28 packets transmitted, 28 received, 0% packet loss, time 27006ms
rtt min/avg/max/mdev = 134.869/135.855/136.683/0.569 ms

aaronjg commented 3 years ago

I just followed the instructions from here: https://openwrt.org/docs/guide-user/network/ipv6/ipv6.nat6 and tested it on a clean install of 19.07.5 and openwrt/openwrt@55e23f2, it worked fine for me on both. No marking on the packets on the "rule start" log line.

# uname -r
4.14.209

$ ping -c4  ipv6.google.com
PING ipv6.google.com(iad23s69-in-x0e.1e100.net (2607:f8b0:4004:815::200e)) 56 data bytes
64 bytes from iad23s69-in-x0e.1e100.net (2607:f8b0:4004:815::200e): icmp_seq=1 ttl=113 time=21.3 ms
64 bytes from iad23s69-in-x0e.1e100.net (2607:f8b0:4004:815::200e): icmp_seq=2 ttl=113 time=18.7 ms
64 bytes from iad23s69-in-x0e.1e100.net (2607:f8b0:4004:815::200e): icmp_seq=3 ttl=113 time=26.8 ms
64 bytes from iad23s69-in-x0e.1e100.net (2607:f8b0:4004:815::200e): icmp_seq=4 ttl=113 time=20.2 ms

--- 2607:f8b0:4004:806::200e ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3000ms
rtt min/avg/max/mdev = 667.986/674.297/681.094/4.641 ms

jamesmacwhite commented 3 years ago

Well that's interesting. What is wrong with my setup then. I dropped all rules except the default. Is it the policy wan_wanb_wanc potentially wrong?

@wackejohn are you saying you also had the problem before adding NAT6 into fw3 directly?

wackejohn commented 3 years ago

@jamesmacwhite

@wackejohn are you saying you also had the problem before adding NAT6 into fw3 directly?

Nope, but the NAT6 script on openwrt wiki was very buggy, mwan3 rules (ipv6) would become invalid when the ipv6 interfaces down and up, I had to reload the firewall manually, then the rules would work again.

jamesmacwhite commented 3 years ago

@wackejohn Good to know. I think fw3 and NAT6 has been broken for a while, which is why that NAT6 helper script exists. If those patches were merged into master, it would remove the need for it entirely, but it looks like at least two of them have existed since 2015 and haven't been merged, hopefully NAT6 support can be added directly someday.

I'm now at a loss what specifically in my setup is causing the behaviour I see though.

aaronjg commented 3 years ago

@jamesmacwhite - I'm not sure what is broken. My setup was a bit simpler, rather than three IPv6 interfaces, I just had two and neither were tunnel interfaces, so perhaps that is part of the problem. But I believe @wackejohn has been using tunnel interfaces and he seems to not have an issue.

Is there any way you can simplify your setup further to home in on the problem?

jamesmacwhite commented 3 years ago

@aaronjg I think I'm going to have to simplify it at some point to dig into it. Given it looks like it's something specific in my configuration now, if you couldn't replicate the issue on a clean 19.07.5 image with NAT6.

If my providers themselves would hurry up and deploy IPv6, it would probably be less complicated, not having to use 6in4, L2TP etc, here's to 2021 being the year of IPv6! (probably not!).

jamesmacwhite commented 3 years ago

It looks like it's something specific with L2TP. I applied the IPv6 routing table fixes and tested it again. If I disable the L2TP interface, 6in4 is the next policy and it works fine, no issues what so ever. So likely there's an issue with L2TP possibly upstream. I tested the simplified NAT6 script, but I don't think it made a difference as I could still get the 50% loss behaviour with L2TP with the simplified version in use, although I might just stick with that anyway.

aaronjg commented 3 years ago

Interesting. It wouldn't be the first bug in L2TP that we've hit related to packet marking. I reached out to A&A to ask them to reactivate my developer account so I can continue to debug further and test if this is resolved on the snapshot kernel.

jamesmacwhite commented 3 years ago

Thank you! That is very much appreciated, it would be interesting to know if the behaviour happens on snapshot, but it certainly isn't the first L2TP issue mwan3 has highlighted before.

I've updated the original issue title to mention L2TP, given it looks like it is specific to this protocol.

jamesmacwhite commented 3 years ago

@aaronjg Did you happen to get your L2TP tunnel from AAISP reactivated to see if this could be replicated at all? I'm curious to know if the latest kernel in 21.02 has this issue. I can't really test this without breaking my network. 21.02 isn't stable yet but I'd imagine it will be after a few more RC releases. Due to the DSA changes, I will have to reset the interfaces and network config in anyway, but don't really want to run it just yet.

jamesmacwhite commented 2 years ago

Unfortunately, the same behaviour happens on 21.02. I've finally made the jump and done all the DSA conversion.

My L2TP IPv6 is timing out on a LAN client every other reply, but it is not happening when pinging with the router directly

Pinging ipv6.l.google.com [2a00:1450:4009:81d::200e] with 32 bytes of data:
Reply from 2a00:1450:4009:81d::200e: time=21ms
Request timed out.
Reply from 2a00:1450:4009:81d::200e: time=24ms
Request timed out.
Reply from 2a00:1450:4009:81d::200e: time=28ms
Request timed out.
Reply from 2a00:1450:4009:81d::200e: time=21ms
Request timed out.
Reply from 2a00:1450:4009:81d::200e: time=27ms

I'm now thinking there is something in 2.10.x compared to 2.8.x that is causing this, but it may be specific to L2TP.

Works fine on 19.07.x with mwan3 2.8.x Doesn't work on 19.07.x with mwan3 2.10.x Doesn't work on 21.02.2 with mwan3 2.10.x

For now, I'm temporarily using the 2.8 package on 21.02, which is probably a bit edgy, but I've gone this far in terms of configuring everything for DSA so I'm going to stick with it.