search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.01k stars 3.48k forks source link

Mwan3track not working properly on wireguard interface #10712

Closed wackejohn closed 4 years ago

wackejohn commented 4 years ago

Maintainer: @feckert Environment: OpenWrt SNAPSHOT, r11625-a512123a4b Mwan3 version: 2.8.2-2 Description: I'm using mwan3 and wireguard as client with latest openwrt snapshot. And it seemed that the issue was caused by mwan3 itself, when mwan3 was running ,the ping -I $Device not working, and then mwan3 would always mark the wireguard interface down, but the wireguard interface actually working... Below is the test resault:

root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5233ms

root@HOME-Router:~# mwan3 stop
root@HOME-Router:~# mwan3 status
Interface status:
 interface wan1 is offline and tracking is down
 interface wan1_6 is offline and tracking is down
 interface wan2 is offline and tracking is down
 interface wan2_6 is offline and tracking is down
 interface wan3 is offline and tracking is down
 interface wan3_6 is offline and tracking is down
 interface wan4 is offline and tracking is down
 interface wan4_6 is offline and tracking is down
 interface wana is offline and tracking is down
 interface wana_6 is offline and tracking is down
 interface wanb is offline and tracking is down
 interface wanb_6 is offline and tracking is down

Current ipv4 policies:

Current ipv6 policies:

Directly connected ipv4 networks:

Directly connected ipv6 networks:

Active ipv4 user rules:

Active ipv6 user rules:

root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=58 time=136 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 5 received, 16.6667% packet loss, time 5007ms
rtt min/avg/max/mdev = 134.652/135.242/136.296/0.592 ms
root@HOME-Router:~# mwan3 restart
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7247ms

And there is a discuss on the forum: https://forum.openwrt.org/t/mwan3track-not-working-properly-on-wireguard/49557

wackejohn commented 4 years ago

@feckert Any update about this issue?

feckert commented 4 years ago

Sorry I have not further explored the subject. But the errors 

RTNETLINK answers: File exists
uci: Entry not found

are strange. First of all we should investigate why this message are shown during mwan3 restart.

wackejohn commented 4 years ago

Sorry I have not further explored the subject. But the errors

RTNETLINK answers: File exists
uci: Entry not found

are strange. First of all we should investigate why this message are shown during mwan3 restart.

These errors are always there, maybe it is caused by my mwan3 and firewall configs. Is there any thing I can do for these errors?

wackejohn commented 4 years ago

@feckert I'm trying to debug myself, and I've found that the mwan3_set_general_iptables() in lib/mwan3/mwan3.sh caused ping failure.

        if ! $IPT -S OUTPUT | grep mwan3_hook &> /dev/null; then
            $IPT -A OUTPUT -j mwan3_hook
        fi

I'm not goot at iptables, is there any way to get the ping working with wireguard interfaces?

feckert commented 4 years ago

I have seen that you do not use the ping from busybox. I think this has everything to do with the interface binding that doesn't work by ip-utils, because the version is over 10 years old. In the pullrequest https://github.com/openwrt/packages/pull/12229 from @aaronjg there are some changes that would fix this. Please give them a try. And if it does not work please reopen this Issue.

wackejohn commented 4 years ago

@feckert Thanks for your reply. I've tried pullrequest #12229 , but the ping still not working (both from busybox and new ip-utils). 

BusyBox v1.31.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r13398-48e3449945
 -----------------------------------------------------
root@HOME-Router_CZ:~# /bin/ping
BusyBox v1.31.1 () multi-call binary.

Usage: ping [OPTIONS] HOST

Send ICMP ECHO_REQUEST packets to network hosts

        -4,-6           Force IP or IPv6 name resolution
        -c CNT          Send only CNT pings
        -s SIZE         Send SIZE data bytes in packets (default 56)
        -i SECS         Interval
        -A              Ping as soon as reply is recevied
        -t TTL          Set TTL
        -I IFACE/IP     Source interface or IP address
        -W SEC          Seconds to wait for the first response (default 10)
                        (after all -c CNT packets are sent)
        -w SEC          Seconds until ping exits (default:infinite)
                        (can exit earlier with -c CNT)
        -q              Quiet, only display output at start
                        and when finished
        -p HEXBYTE      Pattern to use for payload
root@HOME-Router_CZ:~# /usr/bin/ping -V
ping from iputils reboot-13398-g48e3449945+
root@HOME-Router_CZ:~# /bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_CZ:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
7 packets transmitted, 0 received, 100% packet loss, time 6166ms
aaronjg commented 4 years ago

Can you confirm that busybox ping works when mwan3 is not running?

If that doesn't work, can you provide some additional diagnostics:

iptables --table mangle -I POSTROUTING 1 -d 1.1.1.1 -j LOG --log-prefix "postroute start "
iptables --table mangle -A POSTROUTING  -d 1.1.1.1 -j LOG --log-prefix "postroute end  " 
iptables --table mangle -I OUTPUT 1 -d 1.1.1.1 -j LOG --log-prefix "output start "
iptables --table mangle -A OUTPUT  -d 1.1.1.1 -j LOG --log-prefix "output end "
wackejohn commented 4 years ago

@aaronjg The busybox ping works when I stop the mwan3:

root@HOME-Router_CZ:~# mwan3 stop
root@HOME-Router_CZ:~# mwan3 interfaces
Interface status:
 interface wan1 is offline and tracking is down
 interface wan1_6 is offline and tracking is down
 interface wan2 is offline and tracking is down
 interface wan2_6 is offline and tracking is down
 interface wan3 is offline and tracking is down
 interface wan3_6 is offline and tracking is down
 interface wan4 is offline and tracking is down
 interface wan4_6 is offline and tracking is down
 interface wana is offline and tracking is down
 interface wana_6 is offline and tracking is down
 interface wanb is offline and tracking is down
 interface wanb_6 is offline and tracking is down

root@HOME-Router_CZ:~# /bin/ping
BusyBox v1.31.1 () multi-call binary.

Usage: ping [OPTIONS] HOST

Send ICMP ECHO_REQUEST packets to network hosts

        -4,-6           Force IP or IPv6 name resolution
        -c CNT          Send only CNT pings
        -s SIZE         Send SIZE data bytes in packets (default 56)
        -i SECS         Interval
        -A              Ping as soon as reply is recevied
        -t TTL          Set TTL
        -I IFACE/IP     Source interface or IP address
        -W SEC          Seconds to wait for the first response (default 10)
                        (after all -c CNT packets are sent)
        -w SEC          Seconds until ping exits (default:infinite)
                        (can exit earlier with -c CNT)
        -q              Quiet, only display output at start
                        and when finished
        -p HEXBYTE      Pattern to use for payload
root@HOME-Router_CZ:~# /bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=58 time=136.848 ms
64 bytes from 8.8.8.8: seq=1 ttl=58 time=136.043 ms
64 bytes from 8.8.8.8: seq=2 ttl=58 time=136.566 ms
64 bytes from 8.8.8.8: seq=3 ttl=58 time=137.378 ms
64 bytes from 8.8.8.8: seq=4 ttl=58 time=136.683 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 136.043/136.703/137.378 ms
root@HOME-Router_CZ:~# /bin/ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=55 time=141.829 ms
64 bytes from 8.8.8.8: seq=1 ttl=55 time=142.050 ms
64 bytes from 8.8.8.8: seq=2 ttl=55 time=141.761 ms
64 bytes from 8.8.8.8: seq=3 ttl=55 time=141.551 ms
64 bytes from 8.8.8.8: seq=4 ttl=55 time=141.491 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 141.491/141.736/142.050 ms

And the output when mwan3 was running:

root@HOME-Router_CZ:~# ip rule list
0:      from all lookup local
1001:   from all iif pppoe-wan1 lookup 1
1003:   from all iif pppoe-wan2 lookup 3
1005:   from all iif pppoe-wan3 lookup 5
1007:   from all iif pppoe-wan4 lookup 7
1009:   from all iif wana lookup 9
1011:   from all iif wanb lookup 11
2001:   from all fwmark 0x100/0x3f00 lookup 1
2003:   from all fwmark 0x300/0x3f00 lookup 3
2005:   from all fwmark 0x500/0x3f00 lookup 5
2007:   from all fwmark 0x700/0x3f00 lookup 7
2009:   from all fwmark 0x900/0x3f00 lookup 9
2011:   from all fwmark 0xb00/0x3f00 lookup 11
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
32766:  from all lookup main
32767:  from all lookup default
root@HOME-Router_CZ:~# iptables --table mangle -I POSTROUTING 1 -d 8.8.8.8 -j LOG --log-prefix "postroute start "
root@HOME-Router_CZ:~# iptables --table mangle -A POSTROUTING  -d 8.8.8.8 -j LOG --log-prefix "postroute end  " 
root@HOME-Router_CZ:~# iptables --table mangle -I OUTPUT 1 -d 8.8.8.8 -j LOG --log-prefix "output start "
root@HOME-Router_CZ:~# iptables --table mangle -A OUTPUT  -d 8.8.8.8 -j LOG --log-prefix "output end "
root@HOME-Router_CZ:~# /bin/ping -c1 -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_CZ:~# logread |grep "postroute start"
Sat May 30 08:40:14 2020 kern.warn kernel: [34815.895080] postroute start IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=49580 DF PROTO=ICMP TYPE=8 CODE=0 ID=37655 SEQ=0 MARK=0x900
root@HOME-Router_CZ:~# logread |grep "postroute end"
Sat May 30 08:40:14 2020 kern.warn kernel: [34815.910087] postroute end  IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=49580 DF PROTO=ICMP TYPE=8 CODE=0 ID=37655 SEQ=0 MARK=0x900
root@HOME-Router_CZ:~# logread |grep "output start"
Sat May 30 08:40:14 2020 kern.warn kernel: [34815.866656] output start IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=49580 DF PROTO=ICMP TYPE=8 CODE=0 ID=37655 SEQ=0
root@HOME-Router_CZ:~# logread |grep "output end"
Sat May 30 08:40:14 2020 kern.warn kernel: [34815.880484] output end IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=49580 DF PROTO=ICMP TYPE=8 CODE=0 ID=37655 SEQ=0 MARK=0x900
aaronjg commented 4 years ago

Is that the correct src ip? It looks like it is a wan, but has a private IP address.

Can you share the output of ip route list and ip route list table 9

I am wondering if your VPN set up two half internet routes rather than one default route. Those would then be copied over to the other routing tables and could cause issues.

wackejohn commented 4 years ago

@aaronjg wana and wanb were the wireguard vpn and worked as internet connections, so the src ip was correct. The output:

root@HOME-Router_CZ:~# ip route list
default via 180.116.118.1 dev pppoe-wan1 proto static metric 10 
default via 114.228.153.1 dev pppoe-wan2 proto static metric 15 
default via 223.66.36.1 dev pppoe-wan3 proto static metric 20 
default via 183.213.154.1 dev pppoe-wan4 proto static metric 25 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.21 
180.116.118.1 dev pppoe-wan1 proto kernel scope link src 180.116.118.230 
183.213.154.1 dev pppoe-wan4 proto kernel scope link src 183.213.155.69 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.120.0/24 dev wana proto static scope link metric 30 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
223.66.36.1 dev pppoe-wan3 proto kernel scope link src 223.66.36.129 
root@HOME-Router_CZ:~# ip route list table 9
default dev wana scope link metric 30 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.21 
180.116.118.1 dev pppoe-wan1 proto kernel scope link src 180.116.118.230 
183.213.154.1 dev pppoe-wan4 proto kernel scope link src 183.213.155.69 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.120.0/24 dev wana proto static scope link metric 30 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
223.66.36.1 dev pppoe-wan3 proto kernel scope link src 223.66.36.129
aaronjg commented 4 years ago

It looks like you have no routes beyond the subnet for the wireguard interface. If you don't have a default route, then mwan3 isn't going to be able to copy anything over into the relevant table.

I think you need to configure your wireguard to add the default routes, or do it manually after the link comes up.

I'm curious how it is working without mwan3 enabled. Can you turn mwan3 off and post the same results from the iptables logs?

wackejohn commented 4 years ago

The log when mwan3 stoped:

root@HOME-Router_CZ:~# /bin/ping -c1 -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=58 time=193.008 ms

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 193.008/193.008/193.008 ms
root@HOME-Router_CZ:~# logread |grep "postroute start"
Sat May 30 10:14:51 2020 kern.warn kernel: [40492.986831] postroute start IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5275 DF PROTO=ICMP TYPE=8 CODE=0 ID=18496 SEQ=0
root@HOME-Router_CZ:~# logread |grep "postroute end"
Sat May 30 10:14:51 2020 kern.warn kernel: [40493.000785] postroute end  IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5275 DF PROTO=ICMP TYPE=8 CODE=0 ID=18496 SEQ=0
root@HOME-Router_CZ:~# logread |grep "output start"
Sat May 30 10:14:51 2020 kern.warn kernel: [40492.959591] output start IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5275 DF PROTO=ICMP TYPE=8 CODE=0 ID=18496 SEQ=0
root@HOME-Router_CZ:~# logread |grep "output end"
Sat May 30 10:14:51 2020 kern.warn kernel: [40492.973292] output end IN= OUT=wana SRC=192.168.120.2 DST=8.8.8.8 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=5275 DF PROTO=ICMP TYPE=8 CODE=0 ID=18496 SEQ=0

And how to add the default route for wireguard interface?

aaronjg commented 4 years ago

Sorry, I don't use wireguard so I can't help you there. With OtherVPNs, the routes are often pushed to you and the VPN client then sets them up. You could do it manually with ip route add ...

wackejohn commented 4 years ago

I think it's not about the default route thing, I've add the defalut route for wireguard interface, and restarted the mwan3, but the ping still not working.

BusyBox v1.31.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r13398-48e3449945
 -----------------------------------------------------
root@HOME-Router_CZ:~# ip route list
default via 180.116.118.1 dev pppoe-wan1 proto static metric 10 
default via 114.228.153.1 dev pppoe-wan2 proto static metric 15 
default via 223.66.36.1 dev pppoe-wan3 proto static metric 20 
default via 183.213.154.1 dev pppoe-wan4 proto static metric 25 
default via 192.168.120.1 dev wana metric 30 
default via 192.168.150.1 dev wanb metric 40 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.21 
180.116.118.1 dev pppoe-wan1 proto kernel scope link src 180.116.118.230 
183.213.154.1 dev pppoe-wan4 proto kernel scope link src 183.213.155.69 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.120.0/24 dev wana proto static scope link metric 30 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
223.66.36.1 dev pppoe-wan3 proto kernel scope link src 223.66.36.129 
root@HOME-Router_CZ:~# /bin/ping -c1 -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

--- 8.8.8.8 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
wackejohn commented 4 years ago

@feckert Can you reopen this issue for me please? thanks.

aaronjg commented 4 years ago

You may also need a static route to the wireguard endpoint in your table. It does not look like the wireguard routes were set up correctly.

Also, it looks like you are running a development build. Can you try it with the latest stable release?

wackejohn commented 4 years ago

@aaronjg The wireguard endpoint route was deleted and it's working well without the static out, but the mwan3track not working with or without the static route. My device running the development build for years, so it's inconvenient to switch to stable release, but i think it will be the same situation.

aaronjg commented 4 years ago

@wackejohn I just ran into this issue myself. It appears that something is handled differently when the outgoing interface is 192.168.X.X.

Can you try applying this commit, and see if it fixes things for you?

https://github.com/aaronjg/openwrt-packages/commit/3f1152b46da5e66254b7aee234877d2b6d4f0fb3

wackejohn commented 4 years ago

@aaronjg I've tried the commit, but still no lucky...

root@HOME-Router_HB:~# /bin/ping
BusyBox v1.31.1 () multi-call binary.

Usage: ping [OPTIONS] HOST

Send ICMP ECHO_REQUEST packets to network hosts

        -4,-6           Force IP or IPv6 name resolution
        -c CNT          Send only CNT pings
        -s SIZE         Send SIZE data bytes in packets (default 56)
        -i SECS         Interval
        -A              Ping as soon as reply is recevied
        -t TTL          Set TTL
        -I IFACE/IP     Source interface or IP address
        -W SEC          Seconds to wait for the first response (default 10)
                        (after all -c CNT packets are sent)
        -w SEC          Seconds until ping exits (default:infinite)
                        (can exit earlier with -c CNT)
        -q              Quiet, only display output at start
                        and when finished
        -p HEXBYTE      Pattern to use for payload
root@HOME-Router_HB:~# /bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_HB:~# /bin/ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=1 ttl=116 time=37.902 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=37.761 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=37.525 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=37.710 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=38.157 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 5 packets received, 44% packet loss
root@HOME-Router_HB:~# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 38 byte packets
 1  192.168.120.1 (192.168.120.1)  135.816 ms  138.259 ms  138.762 ms
 2  172.22.61.200 (172.22.61.200)  154.769 ms  152.138 ms  154.381 ms
 3  72.14.222.37 (72.14.222.37)  146.069 ms  139.710 ms  146.242 ms
 4  108.170.247.225 (108.170.247.225)  138.113 ms  108.170.247.129 (108.170.247.129)  143.132 ms  108.170.247.225 (108.170.247.225)  182.618 ms
 5  209.85.143.255 (209.85.143.255)  198.962 ms  66.249.94.217 (66.249.94.217)  254.052 ms  142.250.226.109 (142.250.226.109)  224.975 ms
 6  dns.google (8.8.8.8)  141.630 ms  141.739 ms  146.930 ms

And it seemed that the icmp packages didn't pass through the vpn.

aaronjg commented 4 years ago

So, with the trace route it looks like the packets are going through your vpn (192.168.120.1) when you don’t specify the interface, but not when you do? Looks like the issue I ran into is different from the issue I had.

wackejohn commented 4 years ago

@aaronjg As my mwan3 config:

config rule 'AntiGFW_DNS'
        option proto 'all'
        option dest_ip '8.8.8.8,8.8.4.4,208.67.222.222,208.67.220.220,199.91.73.222,178.79.131.110'
        option sticky '0'
        option family 'ipv4'
        option use_policy 'wana_wanb'

the icmp packets should pass through the vpn (wana or wanb) wether specify the interface or not.

aaronjg commented 4 years ago

yes, but there are different rules that are applied to route them out of the interface when you specify the interface vs when you don't.

I am surprised that they are being blocked with the new patch. That should make the ICMP echo packets skip the mwan3 rules entirely when you are specifying the bound interface.

What kernel version are you using? I'll do the best I can to help, but I may not be able to reproduce since you are on the snapshot...

A few more diagnostics - can you repeat the firewall logging with the new patch? I want to see if these packets are somehow still getting marked.

Can you try pinging an IP , eg 1.1.1.1, not covered by your mwan3 rule?

What is the output of: ip -4 addr ls wana ipset list mwan3_source_v4 iptables --table mangle -S mwan3_hook

jamesmacwhite commented 4 years ago

So, not to try and create noise here, but I recently switched from OpenVPN to Wireguard with mwan3, evaluating Wireguard vs OpenVPN, I found the IPv6 side was a bit off, but the latest changes from @aaronjg, fixes those.

I have two Wireguard VPN interfaces as clients load balanced with Mullvad and it works OK. I'm on 19.07.3 kernel 4.14.180 on a Linksys WRT3200ACM. Currently running mwan3 2.8.6, but I'm aware you've tried the changes and it hasn't worked.

Have you tried stripping back the mwan3 config to a very minimal setup?

aaronjg commented 4 years ago

James, not noise at all. Thanks for sharing your experience. What sort of IPv4 address are you getting from Mullvad? I wonder if some of john’s problems is that he is getting a private, 192.168.X.x address from his vpn.

On Tuesday, June 16, 2020, James White notifications@github.com wrote:

So, not to try and create noise here, but I recently switched from OpenVPN to Wireguard with mwan3, evaluating Wireguard vs OpenVPN, I found the IPv6 side was a bit off, but the latest changes from @aaronjg https://github.com/aaronjg, fixes those.

I have two Wireguard VPN interfaces as clients load balanced with Mullvad and it works OK. I'm on 19.07.3 on a Linksys WRT3200ACM. Currently running 2.8.6, but I'm aware you've tried the changes and it hasn't worked.

Have you tried stripping back the mwan3 config to a very minimal setup?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/10712#issuecomment-644949248, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHD3CBW5ZUNCIK3PYH5NVLRW65LPANCNFSM4JXG4K5Q .

-- Sent from Gmail Mobile

jamesmacwhite commented 4 years ago

@aaronjg Mullvad provides an IPv4 in the 10.0.0.0/8 range and a single /128 fc00::/7 ULA as they use NAT66.

In order to have IPv4 and IPv6 working with mwan3, I have a wg6 and wgb6 as an alias to the main wg and wgb interfaces with the /128 ULA configured as a static, so I can have both IPv4 and IPv6 control. As with anything Wireguard currently, you always get a static IPv4 and IPv6 based on your private/public key.

The IPv6 side seems unrelated in this case, because it looks like @wackejohn can't even use the Wireguard inferface once mwan3 is enabled, but in my case:

root@linksys-wrt3200acm:~# traceroute -i wg openwrt.org
traceroute to openwrt.org (139.59.209.225), 30 hops max, 38 byte packets
 1  10.64.0.1 (10.64.0.1)  20.292 ms  25.235 ms  15.759 ms
 2  vlan817.bb2.lon7.uk.m247.com (185.200.118.99)  17.486 ms  15.898 ms  16.395 ms
 3  37.120.220.58 (37.120.220.58)  18.236 ms  21.289 ms  48.244 ms
 4  xe-2-0-0-0.core1.lon2.uk.m247.com (176.10.82.158)  19.059 ms  xe-2-2-2-0.core1.lon2.uk.m247.com (83.97.21.236)  24.275 ms  xe-2-0-0-0.core1.lon2.uk.m247.com (176.10.82.158)  20.991 ms
 5  89.44.212.108 (89.44.212.108)  38.722 ms  89.44.212.110 (89.44.212.110)  31.225 ms  89.44.212.4 (89.44.212.4)  35.174 ms
 6  37.120.128.253 (37.120.128.253)  29.299 ms  32.342 ms  27.820 ms
 7  fra2-edge1.digitalocean.com (80.81.195.151)  31.283 ms  33.836 ms  35.757 ms
 8  *  *  *
 9  *  *  *
10  wiki-01.infra.openwrt.org (139.59.209.225)  42.591 ms  31.464 ms  32.306 ms
root@linksys-wrt3200acm:~# traceroute -i wgb openwrt.org
traceroute to openwrt.org (139.59.209.225), 30 hops max, 38 byte packets
 1  10.64.0.1 (10.64.0.1)  31.416 ms  22.988 ms  33.543 ms
 2  vlan74.xs1a.man4.uk.m247.com (89.238.183.58)  24.190 ms  23.847 ms  23.517 ms
 3  eth-1-0.core-dc1-agg1.man4.uk.m247.com (77.243.185.82)  23.718 ms  22.940 ms  23.778 ms
 4  xe-1-1-1-0.core1.man4.uk.m247.com (77.243.185.66)  24.027 ms  24.940 ms  22.331 ms
 5  te-12-5-0.core-dc2.man4.uk.m247.com (83.97.21.151)  43.766 ms  te-9-3-0.bb1.lon1.uk.m247.com (212.103.51.17)  45.725 ms  te-13-4-0.core-dc2.man4.uk.m247.com (77.243.176.47)  41.174 ms
 6  xe-3-3-1-0.core1.lon2.uk.m247.com (193.9.115.143)  31.692 ms  xe-3-0-2-0.core1.lon2.uk.m247.com (77.243.185.215)  32.758 ms  xe-5-0-0-0.core1.lon2.uk.m247.com (176.10.82.22)  30.290 ms
 7  89.44.212.110 (89.44.212.110)  38.744 ms  xe-4-0-1-0.core1.fra2.de.m247.com (193.9.115.137)  46.178 ms  46.206 ms
 8  37.120.128.253 (37.120.128.253)  41.575 ms  40.368 ms  41.920 ms
 9  *  *  *
10  *  *  *
11  *  *  *
12  wiki-01.infra.openwrt.org (139.59.209.225)  42.859 ms  41.151 ms  39.960 ms

Perhaps it is related to the 192.168.0.0 range.

wackejohn commented 4 years ago

@aaronjg So, I've changed the vpn address to 10.10.1.0/24, but it's still not working.

root@HOME-Router_HB:~# ip -4 addr ls wana
27: wana: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1300 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 10.10.1.3/24 brd 10.10.1.255 scope global wana
       valid_lft forever preferred_lft forever
root@HOME-Router_HB:~# ipset list mwan3_source_v4
Name: mwan3_source_v4
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 704
References: 1
Number of entries: 6
Members:
192.168.196.2
10.10.1.3
117.85.168.51
192.168.2.1
58.219.110.163
192.168.150.3
root@HOME-Router_HB:~# iptables --table mangle -S mwan3_hook
-N mwan3_hook
-A mwan3_hook -p icmp -m set --match-set mwan3_source_v4 src -m icmp --icmp-type 8 -j RETURN
-A mwan3_hook -j CONNMARK --restore-mark --nfmask 0xff00 --ctmask 0xff00
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_ifaces_in
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_connected
-A mwan3_hook -m mark --mark 0x0/0xff00 -j mwan3_rules
-A mwan3_hook -j CONNMARK --save-mark --nfmask 0xff00 --ctmask 0xff00
-A mwan3_hook -m mark ! --mark 0xff00/0xff00 -j mwan3_connected
root@HOME-Router_HB:~# /bin/ping 199.85.126.10
PING 199.85.126.10 (199.85.126.10): 56 data bytes
64 bytes from 199.85.126.10: seq=0 ttl=57 time=29.872 ms
64 bytes from 199.85.126.10: seq=1 ttl=57 time=29.876 ms
64 bytes from 199.85.126.10: seq=2 ttl=57 time=30.275 ms
64 bytes from 199.85.126.10: seq=3 ttl=57 time=29.513 ms
^C
--- 199.85.126.10 ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 29.513/29.884/30.275 ms
root@HOME-Router_HB:~# /bin/ping -i wana 199.85.126.10
ping: invalid number 'wana'
root@HOME-Router_HB:~# /bin/ping -I wana 199.85.126.10
PING 199.85.126.10 (199.85.126.10): 56 data bytes
^C
--- 199.85.126.10 ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_HB:~# traceroute -i wana 199.85.126.10
traceroute to 199.85.126.10 (199.85.126.10), 30 hops max, 38 byte packets
 1  *  *  *
 2  *  *  *
 3  *  *  *
 4  *  *  *
 5  *  *  *
 6^C

My openwrt version:


BusyBox v1.31.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt SNAPSHOT, r13578-b7a8a54542
 -----------------------------------------------------

Kernel version:

Linux HOME-Router_HB 5.4.45 #0 SMP Mon Jun 15 19:16:18 2020 armv7l GNU/Linux

And the firewall log:

root@HOME-Router_HB:~# iptables --table mangle -I POSTROUTING 1 -d 199.85.126.10 -j LOG --log-prefix "postroute start "
root@HOME-Router_HB:~# iptables --table mangle -A POSTROUTING  -d 199.85.126.10 -j LOG --log-prefix "postroute end  " 
root@HOME-Router_HB:~# iptables --table mangle -I OUTPUT 1 -d 199.85.126.10 -j LOG --log-prefix "output start "
root@HOME-Router_HB:~# iptables --table mangle -A OUTPUT  -d 199.85.126.10 -j LOG --log-prefix "output end "
root@HOME-Router_HB:~# /bin/ping -c1 -I wana 199.85.126.10
PING 199.85.126.10 (199.85.126.10): 56 data bytes
^C
--- 199.85.126.10 ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_HB:~# logread |grep "postroute start"
Fri Jun 19 08:30:37 2020 kern.warn kernel: [252543.234957] postroute start IN= OUT=wana SRC=10.10.1.3 DST=199.85.126.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32357 DF PROTO=ICMP TYPE=8 CODE=0 ID=18558 SEQ=0
root@HOME-Router_HB:~# logread |grep "postroute end"
Fri Jun 19 08:30:37 2020 kern.warn kernel: [252543.249298] postroute end  IN= OUT=wana SRC=10.10.1.3 DST=199.85.126.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32357 DF PROTO=ICMP TYPE=8 CODE=0 ID=18558 SEQ=0
root@HOME-Router_HB:~# logread |grep "output start"
Fri Jun 19 08:30:37 2020 kern.warn kernel: [252543.206892] output start IN= OUT=wana SRC=10.10.1.3 DST=199.85.126.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32357 DF PROTO=ICMP TYPE=8 CODE=0 ID=18558 SEQ=0
root@HOME-Router_HB:~# logread |grep "output end"
Fri Jun 19 08:30:37 2020 kern.warn kernel: [252543.220978] output end IN= OUT=wana SRC=10.10.1.3 DST=199.85.126.10 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32357 DF PROTO=ICMP TYPE=8 CODE=0 ID=18558 SEQ=0
root@HOME-Router_HB:~#
jamesmacwhite commented 4 years ago

@wackejohn My only thoughts would be the fact you're on the snapshot builds and the kernel branch is different 4.x vs 5.x. I'll happy provide any configuration of the Wireguard clients and network config I have setup to see if you find anything different that jumps out, but that's the only area I can see at that moment that could explain the difference in behaviour.

aaronjg commented 4 years ago

If private/public address are handled differently, changing the ip address to 10.x.x.x likely won't affect anything.

After applying the patch, the firewall log looks essentially the same with and without mwan3 enabled, so it is surprising that it would work in one case and not the other. Can you use tcpdump to figure out if the packet is actually going out?

Another possibility is that mwan3 is trying to route the raw vpn traffic through the vpn, but it looks like you are only using the vpn for dns? Though one of the ips, didn't look like a dns server 178.79.131.110), so unless that is your wireguard endpoint, or you have additional mwan3 rules, this seems unlikely.

Other than that, perhaps trying a 4.x kernel as @jamesmacwhite suggests could fix it. If so, it would be good to know so that the issue can be fixed before mainline moves to 5.X.

wackejohn commented 4 years ago

@aaronjg I've tried the latest stable release on my another router, and seemed that the issue didn't exist.

BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.3, r11063-85e04e9f46
 -----------------------------------------------------
root@OpenWrt:~# uname -a
Linux OpenWrt 4.14.180 #0 Sat May 16 18:32:20 2020 mips GNU/Linux
root@OpenWrt:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=137.162 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=137.205 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=139.210 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=137.581 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=137.172 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 137.162/137.666/139.210 ms

Then I tried port the 5.x kernel from snapshot to stable release, the issue showed up.


BusyBox v1.30.1 () built-in shell (ash)

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 OpenWrt 19.07.3, r11063-85e04e9f46
 -----------------------------------------------------
root@OpenWrt:~# uname -a
Linux OpenWrt 5.4.46 #0 Sat May 16 18:32:20 2020 mips GNU/Linux
root@OpenWrt:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 0 packets received, 100% packet loss

So, it's about the kernel, snapshot was on 4.19.x and now on 5.4.x, how to fix this issue? I want to use the 5.4.x kernel for my mvebu devices.

jamesmacwhite commented 4 years ago

This is good to know. Potentially now it's an OpenWrt bug in core? Needs reporting here: https://bugs.openwrt.org/.

aaronjg commented 4 years ago

@wackejohn Thanks for doing the diagnostics.

With the mainline kernel, were you able to get things to work without my mwan3_source_v4 patch?

Hard to say exactly where the bug is. It could be an issue with the kernel, in which case you could do a git-bisect to try to locate which kernel commit broke everything.

It could also be an issue with mwan3 is working in the 4.14 kernel, but only because there was a bug in the kernel, so the proper solution could still be to patch mwan3.

Can't really say until we figure out the route cause.

Now that you have a working and broken implementation, it will be a bit easier to diagnose.

Can you use iptables logging and tcpdump to figure out exactly where the behaviors diverge?

Unfortunately, I don't have a router that I can run the snapshot build on, so I won't be able to reproduce.

wackejohn commented 4 years ago

@aaronjg

Are the firewall marks, interfaces and outgoing ip address the same at the end of postrouting on both routers?

All the settings are the same on both routers.

using tcpdump are the packets leaving the vpn interface on both routers?

Tcpdump on kernel 5.4.x, the icmp packtes can't leaving the vpn interface.

root@HOME-Router_CZ:~# /bin/ping -I wana 139.59.209.225
PING 139.59.209.225 (139.59.209.225): 56 data bytes
^C
--- 139.59.209.225 ping statistics ---
478 packets transmitted, 0 packets received, 100% packet loss
root@HOME-Router_CZ:~# tcpdump -i wana -nn host 139.59.209.225
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wana, link-type RAW (Raw IP), capture size 262144 bytes
19:36:13.309118 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 27, length 64
19:36:14.309218 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 28, length 64
19:36:15.309335 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 29, length 64
19:36:16.309431 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 30, length 64
19:36:17.309521 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 31, length 64
19:36:18.309609 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 32, length 64
19:36:19.309699 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 33, length 64
19:36:20.309804 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 34, length 64
19:36:21.309922 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 35, length 64
19:36:22.310020 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 36, length 64
19:36:23.310144 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 37, length 64
19:36:24.310263 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 38, length 64
19:36:25.310353 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 39, length 64
19:36:26.310574 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 40, length 64
19:36:27.310679 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 41, length 64
19:36:28.310781 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 42, length 64
19:36:29.310899 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 43, length 64
19:36:30.311003 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 44, length 64
19:36:31.311097 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 45, length 64
19:36:32.311194 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 46, length 64
19:36:33.311280 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 47, length 64
19:36:34.311369 IP 10.10.1.2 > 139.59.209.225: ICMP echo request, id 65311, seq 48, length 64
root@VPS-Server:~# tcpdump -i wg0 -nn host 139.59.209.225
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
^C
0 packets captured
0 packets received by filter
0 packets dropped by kernel

Hard to say exactly where the bug is. It could be an issue with the kernel, in which case you could do a git-bisect to try to locate which kernel commit broke everything.

The snapshot kernel version was from 4.14 directly to 4.19, so it will be a huge work.

aaronjg commented 4 years ago

yeah, git bisect would probably take several kernel flashes, so not the easiest way to diagnose. Let's keep exploring other possibilities.

It looks like the ping packets are being handled properly to go out of the VPN interface, but the VPN traffic is not being routed properly.

Can you add similar iptables logging to track the wireguard packets, and share the outptu? In particular, I want to see if they are getting different firewall marks with the 5.X kernel vs. the 4.X kernel. You'll have to do some other rule to filter them, perhaps based on destination address or port.

Is the ip address of your wireguard server in the mwan3_connected_v4 ipset? (you can check either in the output of ipset list mwan3_connected_v4 or ipset test mwan3_connected_v4 $SERVER_IP

With the mainline kernel, were you able to get things to work without my mwan3_source_v4 patch?

wackejohn commented 4 years ago

@aaronjg

Can you add similar iptables logging to track the wireguard packets, and share the outptu?

The output (strange thing was that the fist ping succeeded):

root@OpenWrt:~# mwan3 interfaces
Interface status:
 interface wan is online and tracking is active
 interface wan_6 is online and tracking is active
 interface wana is online and tracking is not enabled

root@OpenWrt:~# ping -I wana -c 1 openwrt.org
ping: connect: Permission denied
root@OpenWrt:~# /bin/ping -I wana -c 1 openwrt.org
PING openwrt.org (139.59.209.225): 56 data bytes
64 bytes from 139.59.209.225: seq=0 ttl=52 time=545.257 ms

--- openwrt.org ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 545.257/545.257/545.257 ms
root@OpenWrt:~# /bin/ping -I wana -c 1 openwrt.org
PING openwrt.org (139.59.209.225): 56 data bytes

--- openwrt.org ping statistics ---
1 packets transmitted, 0 packets received, 100% packet loss

root@OpenWrt:~# logread |grep "postroute start"
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.142138] postroute startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=1880 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.342776] postroute startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=1889 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.613739] postroute startIN= OUT=wana SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:37:08 2020 kern.warn kernel: [  268.944923] postroute startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=204 TOS=0x00 PREC=0xC0 TTL=64 ID=4435 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.116.118.253 LEN=176 TOS=0x00 PREC=0x00 TTL=54 ID=3942 PROTO=UDP SPT=51888 DPT=36686 LEN=156 ] MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.124283] postroute startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=3389 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.321857] postroute startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3406 PROTO=UDP SPT=36686 DPT=51888 LEN=40 MARK=0x100
root@OpenWrt:~# logread |grep "postroute end"
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.158003] postroute endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=1880 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.358639] postroute endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=1889 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.629166] postroute endIN= OUT=wana SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:37:08 2020 kern.warn kernel: [  268.970377] postroute endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=204 TOS=0x00 PREC=0xC0 TTL=64 ID=4435 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.116.118.253 LEN=176 TOS=0x00 PREC=0x00 TTL=54 ID=3942 PROTO=UDP SPT=51888 DPT=36686 LEN=156 ] MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.140148] postroute endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=3389 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.337545] postroute endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3406 PROTO=UDP SPT=36686 DPT=51888 LEN=40 MARK=0x100
root@OpenWrt:~# logread |grep "output start"
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.111960] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=1880 PROTO=UDP SPT=36686 DPT=51888 LEN=156
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.312617] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=1889 PROTO=UDP SPT=36686 DPT=51888 LEN=136
Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.583575] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136
Sat Jun 27 23:37:08 2020 kern.warn kernel: [  268.895579] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=204 TOS=0x00 PREC=0xC0 TTL=64 ID=4435 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.116.118.253 LEN=176 TOS=0x00 PREC=0x00 TTL=54 ID=3942 PROTO=UDP SPT=51888 DPT=36686 LEN=156 ]
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.094108] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=3389 PROTO=UDP SPT=36686 DPT=51888 LEN=156
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.292050] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3406 PROTO=UDP SPT=36686 DPT=51888 LEN=40
root@OpenWrt:~# logread |grep "output end"
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.126658] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=1880 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:36:52 2020 kern.warn kernel: [  253.327305] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=1889 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.598261] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
Sat Jun 27 23:37:08 2020 kern.warn kernel: [  268.919855] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=204 TOS=0x00 PREC=0xC0 TTL=64 ID=4435 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.116.118.253 LEN=176 TOS=0x00 PREC=0x00 TTL=54 ID=3942 PROTO=UDP SPT=51888 DPT=36686 LEN=156 ] MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.108802] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=3389 PROTO=UDP SPT=36686 DPT=51888 LEN=156 MARK=0x100
Sat Jun 27 23:37:12 2020 kern.warn kernel: [  273.306561] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=3406 PROTO=UDP SPT=36686 DPT=51888 LEN=40 MARK=0x100
root@OpenWrt:~#

My mwan3 config:

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option initial_state 'online'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'

config interface 'wan_6'
        option enabled '1'
        list track_ip '240e:5a::6666'
        list track_ip '240e:5b::6666'
        option family 'ipv6'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wana'
        option family 'ipv4'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option enabled '1'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'

config member 'wan_m1_w1'
        option interface 'wan'
        option metric '1'
        option weight '1'

config member 'wana_m1_w1'
        option interface 'wana'
        option metric '1'
        option weight '1'

config member 'wan_6_m1_w1'
        option interface 'wan_6'
        option metric '1'
        option weight '1'

config policy 'wan_only'
        list use_member 'wan_m1_w1'
        list use_member 'wan_6_m1_w1'

config policy 'wana_only'
        list use_member 'wana_m1_w1'

config policy 'balanced'
        list use_member 'wan_m1_w1'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'balanced'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

Is the ip address of your wireguard server in the mwan3_connected_v4 ipset?

My wireguard server address (173.242.119.141) isn't in the mwan3_connected_v4 ipset:

root@OpenWrt:/etc/config# ipset list mwan3_connected_v4
Name: mwan3_connected_v4
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 1208
References: 1
Number of entries: 15
Members:
180.116.118.253
127.255.255.255
192.168.1.0/24
180.116.118.1
10.10.1.0/24
127.0.0.1
10.10.1.0
10.10.1.5
192.168.1.255
192.168.1.0
192.168.1.1
224.0.0.0/3
127.0.0.0/8
127.0.0.0
10.10.1.255

With the mainline kernel, were you able to get things to work without my mwan3_source_v4 patch?

Acctully, kernel above 4.19, the ping -I won't work with or without your patch.

aaronjg commented 4 years ago

What do you get for: ip route list table 1 this is how it is trying to route out the traffic to your firewall server.

Normally, the vpn server would have a static route and be in the connected ipset list, and ignored. Earlier, you said that route was deleted. Why?

Acctully, kernel above 4.19, the ping -I won't work with or without your patch.

Yes, but what about the earlier kernels, do you need my patch to make that work?

wackejohn commented 4 years ago

Normally, the vpn server would have a static route and be in the connected ipset list, and ignored. Earlier, you said that route was deleted. Why?

Cause I have multi ISPs for internet connection, and I want to use policy to control the wireguard connection with mwan3, so I delete the static route (snapshot version of wireguard have an option "nohostroute" do the same thing).

Yes, but what about the earlier kernels, do you need my patch to make that work?

On earlier kernels (4.14 and below ), the ping -I is working and no need your patch.

aaronjg commented 4 years ago

Can you describe exactly what you are trying to do and post your complete mwan3 configuration and the output of ip route list # for every '#inip rule list` (I think 1,3,5,7 and 9 from your previous post).

Mwan3 needs that route to include it in the mwan3_connected set so that it knows to let the traffic go to the VPN server. Otherwise, it may end up trying to route the traffic that should be to the VPN server through the VPN server, which won't work.

If you post your configuration and what it is exactly you are trying to accomplish, there may be a way to still do that with additional firewall rules. However, this is a bit outside the box of the 'normal' mwan3 setup.

wackejohn commented 4 years ago

@aaronjg

Can you describe exactly what you are trying to do

I'm trying to bypass the ISP firewall through wireguard vpn, including DNS and Routing, and then use mwan3 to achieve the failover policy for two wireguard vpn connections.

My mwan3 config (will be very complicated):

config rule 'AntiGFW_DNS'
        option proto 'all'
        option family 'ipv4'
        option dest_ip '8.8.8.8,8.8.4.4,208.67.222.222,208.67.220.220,199.91.73.222,178.79.131.110'
        option sticky '1'
        option timeout '600'
        option use_policy 'wana_wanb'

config rule 'ChinaNet_DNS'
        option dest_ip '218.4.4.4,218.2.2.2,114.114.114.114,223.5.5.5'
        option family 'ipv4'
        option proto 'all'
        option sticky '1'
        option timeout '600'
        option use_policy 'wan1_wan2'

config rule 'CM_DNS'
        option dest_ip '112.4.0.55,221.131.143.69,114.114.115.115,223.6.6.6'
        option family 'ipv4'
        option proto 'all'
        option sticky '1'
        option timeout '600'
        option use_policy 'wan3_wan4'

config rule 'Clock_VPS'
        option family 'ipv4'
        option proto 'all'
        option dest_ip '119.23.32.83,31.40.214.127,173.242.119.141'
        option use_policy 'wan1_2_default'
        option sticky '0'

config rule 'IPV4_PTlist'
        option proto 'all'
        option family 'ipv4'
        option ipset 'ptlist'
        option sticky '0'
        option use_policy 'wan3_wan4'

config rule 'IPV4_GFW'
        option proto 'all'
        option family 'ipv4'
        option ipset 'gfwlist'
        option timeout '300'
        option sticky '0'
        option use_policy 'wana_wanb'

config rule 'CM_IPV4'
        option proto 'all'
        option family 'ipv4'
        option ipset 'cmcc'
        option timeout '600'
        option sticky '1'
        option use_policy 'wan3_wan4'

config rule 'ChinaNet_IPV4'
        option proto 'all'
        option family 'ipv4'
        option timeout '600'
        option sticky '1'
        option use_policy 'wan1_wan2'
        option ipset 'cnnocmcc'

config rule 'IPV4_WEB'
        option proto 'all'
        option family 'ipv4'
        option sticky '1'
        option use_policy 'wan_1_2_3_4'

config rule 'IPV6_PTlist'
        option proto 'all'
        option family 'ipv6'
        option ipset 'ptlist6'
        option sticky '0'
        option use_policy 'wan3_6_wan4_6'

config rule 'IPV6_GFW'
        option proto 'all'
        option family 'ipv6'
        option sticky '0'
        option ipset 'gfwlist6'
        option use_policy 'wanb_6_wana_6'

config rule 'CM_IPV6'
        option proto 'all'
        option family 'ipv6'
        option ipset 'cmccv6'
        option timeout '600'
        option sticky '1'
        option use_policy 'wan3_6_wan4_6'

config rule 'ChinaNet_IPV6'
        option proto 'all'
        option family 'ipv6'
        option timeout '600'
        option sticky '1'
        option use_policy 'wan1_6_wan2_6'
        option ipset 'cnnocmccv6'

config rule 'IPV6_WEB'
        option proto 'all'
        option family 'ipv6'
        option timeout '600'
        option sticky '1'
        option use_policy 'wan12_6_wan34_6'

config globals 'globals'
        option mmx_mask '0x3F00'
        option rtmon_interval '5'

config interface 'wan1'
        option enabled '1'
        option family 'ipv4'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        list track_ip '223.5.5.5'
        list track_ip '223.6.6.6'
        option size '56'
        option timeout '2'
        option initial_state 'online'
        option track_method 'ping'
        option online_metric '1'
        option count '3'
        option interval '5'
        option reliability '2'
        option failure_interval '3'
        option recovery_interval '3'
        option max_ttl '60'
        option check_quality '0'
        option down '3'
        option up '2'

config interface 'wan1_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option up '2'
        option reliability '2'
        option down '3'
        option size '56'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        list track_ip '240e:5a::6666'
        list track_ip '240e:5b::6666'
        option track_method 'ping'
        option check_quality '0'
        option initial_state 'online'

config interface 'wan2'
        option enabled '1'
        option family 'ipv4'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        list track_ip '223.5.5.5'
        list track_ip '223.6.6.6'
        option size '56'
        option timeout '2'
        option initial_state 'online'
        option track_method 'ping'
        option online_metric '1'
        option count '3'
        option interval '5'
        option reliability '2'
        option failure_interval '3'
        option recovery_interval '3'
        option max_ttl '60'
        option check_quality '0'
        option down '3'
        option up '2'

config interface 'wan2_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option up '2'
        option reliability '2'
        option down '3'
        option size '56'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        list track_ip '240e:5a::6666'
        list track_ip '240e:5b::6666'
        option track_method 'ping'
        option check_quality '0'
        option initial_state 'online'

config interface 'wan3'
        option enabled '1'
        option family 'ipv4'
        list track_ip '112.4.0.55'
        list track_ip '221.131.143.69'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        option up '2'
        option reliability '2'
        option count '3'
        option size '56'
        option timeout '2'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        option initial_state 'online'
        option track_method 'ping'
        option online_metric '2'
        option max_ttl '60'
        option check_quality '0'
        option down '3'

config interface 'wan3_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option up '2'
        option reliability '2'
        option down '3'
        option size '56'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        list track_ip '2409:8020:2000::88'
        list track_ip '2409:8020:2000::8'
        option track_method 'ping'
        option check_quality '0'
        option initial_state 'online'

config interface 'wan4'
        option enabled '1'
        option family 'ipv4'
        list track_ip '112.4.0.55'
        list track_ip '221.131.143.69'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        option up '2'
        option reliability '2'
        option count '3'
        option size '56'
        option timeout '2'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        option initial_state 'online'
        option track_method 'ping'
        option online_metric '2'
        option max_ttl '60'
        option check_quality '0'
        option down '3'

config interface 'wan4_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option up '2'
        option reliability '2'
        option down '3'
        option size '56'
        option interval '5'
        option failure_interval '3'
        option recovery_interval '3'
        list track_ip '2409:8020:2000::88'
        list track_ip '2409:8020:2000::8'
        option track_method 'ping'
        option check_quality '0'
        option initial_state 'online'

config interface 'wana'
        option enabled '1'
        option family 'ipv4'
        option timeout '2'
        option count '3'
        option down '3'
        option failure_interval '3'
        option initial_state 'online'
        option keep_failure_interval '1'
        option reliability '1'
        option interval '5'
        option up '1'
        option recovery_interval '3'
        option track_method 'nping-icmp'
        list track_ip '10.10.1.1'

config interface 'wana_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option down '3'
        option failure_interval '3'
        option recovery_interval '3'
        option initial_state 'online'
        option keep_failure_interval '1'
        option reliability '1'
        option interval '5'
        option up '1'
        list track_ip 'dda1:aaaa:e111::1'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'

config interface 'wanb'
        option enabled '1'
        option family 'ipv4'
        option count '3'
        option timeout '2'
        option down '3'
        option failure_interval '3'
        option recovery_interval '3'
        option initial_state 'online'
        option keep_failure_interval '1'
        option reliability '1'
        option interval '5'
        option up '1'
        list track_ip '192.168.150.1'
        option track_method 'nping-icmp'

config interface 'wanb_6'
        option enabled '1'
        option family 'ipv6'
        option count '3'
        option timeout '2'
        option down '3'
        option failure_interval '3'
        option recovery_interval '3'
        option initial_state 'online'
        option keep_failure_interval '1'
        option reliability '1'
        option interval '5'
        option up '1'
        list track_ip 'ddc1:aaaa:f111::1'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'

config member 'wan1_m1_w1'
        option metric '1'
        option interface 'wan1'
        option weight '1'

config member 'wan1_m2_w1'
        option metric '2'
        option interface 'wan1'
        option weight '1'

config member 'wan1_m3_w1'
        option metric '3'
        option interface 'wan1'
        option weight '1'

config member 'wan1_m4_w1'
        option metric '4'
        option interface 'wan1'
        option weight '1'

config member 'wan2_m1_w1'
        option interface 'wan2'
        option weight '1'
        option metric '1'

config member 'wan2_m2_w1'
        option interface 'wan2'
        option weight '1'
        option metric '2'

config member 'wan2_m3_w1'
        option interface 'wan2'
        option weight '1'
        option metric '3'

config member 'wan2_m4_w1'
        option interface 'wan2'
        option weight '1'
        option metric '4'

config member 'wan3_m1_w1'
        option interface 'wan3'
        option weight '1'
        option metric '1'

config member 'wan3_m2_w1'
        option interface 'wan3'
        option weight '1'
        option metric '2'

config member 'wan3_m3_w1'
        option interface 'wan3'
        option weight '1'
        option metric '3'

config member 'wan3_m4_w1'
        option interface 'wan3'
        option weight '1'
        option metric '4'

config member 'wan4_m1_w1'
        option interface 'wan4'
        option weight '1'
        option metric '1'

config member 'wan4_m2_w1'
        option interface 'wan4'
        option weight '1'
        option metric '2'

config member 'wan4_m3_w1'
        option interface 'wan4'
        option weight '1'
        option metric '3'

config member 'wan4_m4_w1'
        option interface 'wan4'
        option weight '1'
        option metric '4'

config member 'wan1_6_m1_w1'
        option interface 'wan1_6'
        option weight '1'
        option metric '1'

config member 'wan1_6_m2_w1'
        option interface 'wan1_6'
        option weight '1'
        option metric '2'

config member 'wan1_6_m3_w1'
        option interface 'wan1_6'
        option weight '1'
        option metric '3'

config member 'wan1_6_m4_w1'
        option interface 'wan1_6'
        option weight '1'
        option metric '4'

config member 'wan2_6_m1_w1'
        option interface 'wan2_6'
        option weight '1'
        option metric '1'

config member 'wan2_6_m2_w1'
        option interface 'wan2_6'
        option weight '1'
        option metric '2'

config member 'wan2_6_m3_w1'
        option interface 'wan2_6'
        option weight '1'
        option metric '3'

config member 'wan2_6_m4_w1'
        option interface 'wan2_6'
        option weight '1'
        option metric '4'

config member 'wan3_6_m1_w1'
        option interface 'wan3_6'
        option weight '1'
        option metric '1'

config member 'wan3_6_m2_w1'
        option interface 'wan3_6'
        option weight '1'
        option metric '2'

config member 'wan3_6_m3_w1'
        option interface 'wan3_6'
        option weight '1'
        option metric '3'

config member 'wan3_6_m4_w1'
        option interface 'wan3_6'
        option weight '1'
        option metric '4'

config member 'wan4_6_m1_w1'
        option interface 'wan4_6'
        option weight '1'
        option metric '1'

config member 'wan4_6_m2_w1'
        option interface 'wan4_6'
        option weight '1'
        option metric '2'

config member 'wan4_6_m3_w1'
        option interface 'wan4_6'
        option weight '1'
        option metric '3'

config member 'wan4_6_m4_w1'
        option interface 'wan4_6'
        option weight '1'
        option metric '4'

config member 'wana_m1_w1'
        option interface 'wana'
        option metric '1'
        option weight '1'

config member 'wana_m2_w1'
        option interface 'wana'
        option metric '2'
        option weight '1'

config member 'wanb_m1_w1'
        option interface 'wanb'
        option weight '1'
        option metric '1'

config member 'wanb_m2_w1'
        option interface 'wanb'
        option weight '1'
        option metric '2'

config member 'wana_6_m1_w1'
        option interface 'wana_6'
        option weight '1'
        option metric '1'

config member 'wana_6_m2_w1'
        option interface 'wana_6'
        option weight '1'
        option metric '2'

config member 'wanb_6_m1_w1'
        option interface 'wanb_6'
        option weight '1'
        option metric '1'

config member 'wanb_6_m2_w1'
        option interface 'wanb_6'
        option weight '1'
        option metric '2'

config policy 'wan1_only'
        list use_member 'wan1_m1_w1'
        option last_resort 'unreachable'

config policy 'wan2_only'
        list use_member 'wan2_m1_w1'
        option last_resort 'unreachable'

config policy 'wan3_only'
        list use_member 'wan3_m1_w1'
        option last_resort 'unreachable'

config policy 'wan4_only'
        list use_member 'wan4_m1_w1'
        option last_resort 'unreachable'

config policy 'wan1_wan2'
        option last_resort 'unreachable'
        list use_member 'wan1_m1_w1'
        list use_member 'wan2_m2_w1'

config policy 'wan1_2_default'
        option last_resort 'default'
        list use_member 'wan1_m1_w1'
        list use_member 'wan2_m2_w1'

config policy 'wan1_2_wan3_4'
        list use_member 'wan1_m1_w1'
        list use_member 'wan2_m1_w1'
        list use_member 'wan3_m2_w1'
        list use_member 'wan4_m2_w1'
        option last_resort 'unreachable'

config policy 'wan3_4_wan1_2'
        option last_resort 'unreachable'
        list use_member 'wan3_m1_w1'
        list use_member 'wan4_m1_w1'
        list use_member 'wan1_m2_w1'
        list use_member 'wan2_m2_w1'

config policy 'wan3_wan4'
        option last_resort 'unreachable'
        list use_member 'wan3_m1_w1'
        list use_member 'wan4_m2_w1'

config policy 'wan3_4_2_1'
        option last_resort 'unreachable'
        list use_member 'wan3_m1_w1'
        list use_member 'wan4_m2_w1'
        list use_member 'wan2_m3_w1'
        list use_member 'wan1_m4_w1'

config policy 'wan4_3_2_1'
        option last_resort 'unreachable'
        list use_member 'wan4_m1_w1'
        list use_member 'wan3_m2_w1'
        list use_member 'wan2_m3_w1'
        list use_member 'wan1_m4_w1'

config policy 'wan1_2_3_4'
        option last_resort 'default'
        list use_member 'wan1_m1_w1'
        list use_member 'wan2_m2_w1'
        list use_member 'wan3_m3_w1'
        list use_member 'wan4_m4_w1'

config policy 'loadbalanced'
        option last_resort 'unreachable'
        list use_member 'wan4_m1_w1'
        list use_member 'wan3_m1_w1'
        list use_member 'wan2_m1_w1'
        list use_member 'wan1_m1_w1'

config policy 'wan1_6_only'
        list use_member 'wan1_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan2_6_only'
        list use_member 'wan2_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan3_6_only'
        list use_member 'wan3_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan4_6_only'
        list use_member 'wan4_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan1_6_wan2_6'
        list use_member 'wan1_6_m1_w1'
        list use_member 'wan2_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan3_6_wan4_6'
        list use_member 'wan3_6_m1_w1'
        list use_member 'wan4_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wan12_6_wan34_6'
        list use_member 'wan1_6_m1_w1'
        list use_member 'wan2_6_m2_w1'
        list use_member 'wan3_6_m3_w1'
        list use_member 'wan4_6_m4_w1'
        option last_resort 'unreachable'

config policy 'wan34_6_wan12_6'
        list use_member 'wan3_6_m1_w1'
        list use_member 'wan4_6_m1_w1'
        list use_member 'wan1_6_m2_w1'
        list use_member 'wan2_6_m2_w1'
        option last_resort 'unreachable'

config policy 'loadbalanced6'
        option last_resort 'unreachable'
        list use_member 'wan1_6_m1_w1'
        list use_member 'wan2_6_m1_w1'
        list use_member 'wan3_6_m1_w1'
        list use_member 'wan4_6_m1_w1'

config policy 'wana_only'
        list use_member 'wana_m1_w1'
        option last_resort 'unreachable'

config policy 'wanb_only'
        list use_member 'wanb_m2_w1'
        option last_resort 'unreachable'

config policy 'wana_wanb'
        list use_member 'wana_m1_w1'
        list use_member 'wanb_m2_w1'
        option last_resort 'unreachable'

config policy 'wanb_wana'
        list use_member 'wanb_m1_w1'
        list use_member 'wana_m2_w1'
        option last_resort 'unreachable'

config policy 'wana_6_only'
        list use_member 'wana_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wanb_6_only'
        list use_member 'wanb_6_m1_w1'
        option last_resort 'unreachable'

config policy 'wana_6_wanb_6'
        list use_member 'wana_6_m1_w1'
        list use_member 'wanb_6_m2_w1'
        option last_resort 'unreachable'

config policy 'wanb_6_wana_6'
        list use_member 'wanb_6_m1_w1'
        list use_member 'wana_6_m2_w1'
        option last_resort 'unreachable'

As above, I want to control the wireguard connection to go out from wan1 or wan2, but if the wireguard create the static route, the connection will allways go out from the wan1, so I delete the static route.

Ip route list:

default via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
default via 114.228.153.1 dev pppoe-wan2 proto static metric 15 
default via 112.3.84.1 dev pppoe-wan3 proto static metric 20 
default via 112.1.96.1 dev pppoe-wan4 proto static metric 25 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan4 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan3 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110

Ip rule list:

0:      from all lookup local
1001:   from all iif pppoe-wan1 lookup 1
1003:   from all iif pppoe-wan2 lookup 3
1005:   from all iif pppoe-wan3 lookup 5
1007:   from all iif pppoe-wan4 lookup 7
1009:   from all iif wana lookup 9
1011:   from all iif wanb lookup 11
2001:   from all fwmark 0x100/0x3f00 lookup 1
2003:   from all fwmark 0x300/0x3f00 lookup 3
2005:   from all fwmark 0x500/0x3f00 lookup 5
2007:   from all fwmark 0x700/0x3f00 lookup 7
2009:   from all fwmark 0x900/0x3f00 lookup 9
2011:   from all fwmark 0xb00/0x3f00 lookup 11
2061:   from all fwmark 0x3d00/0x3f00 blackhole
2062:   from all fwmark 0x3e00/0x3f00 unreachable
32766:  from all lookup main
32767:  from all lookup default

As above, now I keep the static route of wireguard, but the ping -I is also not working, so it's not about the static route thing.

root@HOME-Router_CZ:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 10.10.1.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4192ms
aaronjg commented 4 years ago

Can you post the ip route list table X for each of the routing tables? When you keep the static route, does the VPN ip address show up in ipset list mwan3_source_v4 ?

Can you also post the routing table when you keep the static route?

wackejohn commented 4 years ago

@aaronjg output:

root@HOME-Router_CZ:~# ip route list table 1
default via 180.115.231.1 dev pppoe-wan1 metric 10 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
root@HOME-Router_CZ:~# ip route list table 3
default via 114.228.153.1 dev pppoe-wan2 metric 15 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
root@HOME-Router_CZ:~# ip route list table 5
default via 112.1.96.1 dev pppoe-wan3 metric 20 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
root@HOME-Router_CZ:~# ip route list table 7
default via 112.3.84.1 dev pppoe-wan4 metric 25 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
root@HOME-Router_CZ:~# ip route list table 9
default dev wana scope link metric 30 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
root@HOME-Router_CZ:~# ip route list table 11
default dev wanb scope link metric 40 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110

Seems that there is no mwan3_source_v4?

root@HOME-Router_CZ:~# ipset list mwan3_source_v4
ipset v7.4: The set with the given name does not exist

routing table:

root@HOME-Router_CZ:~# ip route show
default via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
default via 114.228.153.1 dev pppoe-wan2 proto static metric 15 
default via 112.1.96.1 dev pppoe-wan3 proto static metric 20 
default via 112.3.84.1 dev pppoe-wan4 proto static metric 25 
10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110
aaronjg commented 4 years ago

Seems that there is no mwan3_source_v4?

My mistake - I meant to ask about mwan3_connected_v4

aaronjg commented 4 years ago

Also, could you rerun the diagnostics from this comment with the static route added back in?

https://github.com/openwrt/packages/issues/10712#issuecomment-650578269

There are a couple of weird things one:

This is the weird iptables behavior:

Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.583575] output startIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136

Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.598261] output endIN= OUT=pppoe-wan SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100

Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.613739] postroute startIN= OUT=wana SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100

Sat Jun 27 23:36:56 2020 kern.warn kernel: [  257.629166] postroute endIN= OUT=wana SRC=180.116.118.253 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=2153 PROTO=UDP SPT=36686 DPT=51888 LEN=136 MARK=0x100
wackejohn commented 4 years ago

@aaronjg 

root@HOME-Router_CZ:~# ipset list mwan3_connected_v4
Name: mwan3_connected_v4
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 2832
References: 1
Number of entries: 44
Members:
192.168.1.0/24
173.242.119.141
192.168.150.2
127.0.0.1
114.228.153.129
192.168.195.0/24
112.3.84.1
192.168.195.2
192.168.195.0
10.10.1.255
192.168.1.1
192.168.3.0/24
192.168.150.255
10.10.1.0/24
192.168.195.3
114.228.153.1
192.168.196.3
180.115.231.180
192.168.150.0
192.168.195.255
192.168.195.6
112.3.87.193
127.0.0.0/8
192.168.1.255
10.10.1.0
127.255.255.255
31.40.214.127
112.1.96.1
180.115.231.1
192.168.1.0
192.168.195.1
192.168.196.1
192.168.195.4
192.168.196.0/24
192.168.196.2
192.168.196.0
10.10.1.2
192.168.195.5
127.0.0.0
224.0.0.0/3
192.168.150.0/24
112.1.96.195
192.168.196.255
192.168.2.0/24

As above, the vpn server ip was in the mwan3_connected_v4

Also, could you rerun the diagnostics from this comment with the static route added back in?

Actually, it was another router with minimal configuration for test, please ignore that output.

aaronjg commented 4 years ago

Actually, it was another router with minimal configuration for test, please ignore that output.

I see. It will certainly be easier to debug with the minimal configuration. Can you post the full mwan3 config from the minimal configuration, the routing tables as above from the minimal configuration setup, and rerun the iptables logging now that the static route is in place?

I'm trying to see why the VPN packets are getting a firewall mark of 0x100 - normally these should get marked 0x3f00 and then not match any further rules.

wackejohn commented 4 years ago

@aaronjg The minimal mwan3 config:

config interface 'wan'
        option enabled '1'
        option family 'ipv4'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option initial_state 'online'
        list track_ip '114.114.114.114'
        list track_ip '114.114.115.115'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'

config interface 'wan_6'
        option enabled '1'
        list track_ip '240e:5a::6666'
        list track_ip '240e:5b::6666'
        option family 'ipv6'
        option reliability '2'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'

config interface 'wana'
        option family 'ipv4'
        option reliability '1'
        option count '1'
        option timeout '2'
        option interval '5'
        option down '3'
        option up '8'
        option enabled '1'
        option initial_state 'online'
        option track_method 'ping'
        option size '56'
        option max_ttl '60'
        option check_quality '0'
        option failure_interval '5'
        option recovery_interval '5'

config member 'wan_m1_w1'
        option interface 'wan'
        option metric '1'
        option weight '1'

config member 'wana_m1_w1'
        option interface 'wana'
        option metric '1'
        option weight '1'

config member 'wan_6_m1_w1'
        option interface 'wan_6'
        option metric '1'
        option weight '1'

config policy 'wan_only'
        list use_member 'wan_m1_w1'
        list use_member 'wan_6_m1_w1'

config policy 'wana_only'
        list use_member 'wana_m1_w1'

config policy 'balanced'
        list use_member 'wan_m1_w1'

config rule 'https'
        option sticky '1'
        option dest_port '443'
        option proto 'tcp'
        option use_policy 'balanced'

config rule 'default_rule'
        option dest_ip '0.0.0.0/0'
        option use_policy 'balanced'

Routing table:

root@OpenWrt:~# ip route show
default via 117.85.168.1 dev pppoe-wan proto static 
10.10.1.0/24 dev wana proto static scope link metric 10 
117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 
173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
root@OpenWrt:~# ip route list table 1
default via 117.85.168.1 dev pppoe-wan 
10.10.1.0/24 dev wana proto static scope link metric 10 
117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 
173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
root@OpenWrt:~# ip route list table 3
default dev wana scope link metric 10 
10.10.1.0/24 dev wana proto static scope link metric 10 
117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 
173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

And the iptable log:


[  294.718366] output startIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=50646 PROTO=UDP SPT=57303 DPT=51888 LEN=156 
[  294.732883] output endIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=50646 PROTO=UDP SPT=57303 DPT=51888 LEN=156 MARK=0x3f00 
[  294.748276] postroute startIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=50646 PROTO=UDP SPT=57303 DPT=51888 LEN=156 MARK=0x3f00 
[  294.764055] postroute endIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=50646 PROTO=UDP SPT=57303 DPT=51888 LEN=156 MARK=0x3f00 
[  294.919942] output startIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=50661 PROTO=UDP SPT=57303 DPT=51888 LEN=136 
[  294.934433] output endIN= OUT=pppoe-wan SRC=117.85.170.4 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=50661 PROTO=UDP SPT=57303 DPT=51888 LEN=136 MARK=0x3f00

Seemed that with the static route in place, the firewall mark was 0x3f00 .

aaronjg commented 4 years ago

Okay, so with the static route in place, this is starting to look like a more 'normal' mwan3 config, so it will be easier to figure out teh root cause, rather than get distracted by anamolies of your more complex config.

Is that all of the relevant output from the iptables log? something looks very strange - the first packet appears to be routed correctly out of ppoe-wan, but the second one appears to go through the output routing table, but never make it to the postrouting table. Yet, as far as I can tell, aside from the length, the packets are identical. Very strange.

Without mwan3, what do the packets look like going through the firewall? It also is interesting that the UDP packet length of the first packet is the IP packet length of the second. If the iptable logs look different, could you share the headers of these two UDP packets and the corresponding packets without mwan3?

Also, now that you have the minimal config with the static route, can you check with tcpdump if packets are going through the ppoe interface and the wireguard interface, as you did here: https://github.com/openwrt/packages/issues/10712#issuecomment-648792167

wackejohn commented 4 years ago

@aaronjg Mwan3 was running:

root@OpenWrt:~# iptables --table mangle -I POSTROUTING 1 -d 173.242.119.141 -j LOG --log-prefix "postroute start"
root@OpenWrt:~# iptables --table mangle -A POSTROUTING  -d 173.242.119.141 -j LOG --log-prefix "postroute end"
root@OpenWrt:~# iptables --table mangle -I OUTPUT 1 -d 173.242.119.141 -j LOG --log-prefix "output start"
root@OpenWrt:~# iptables --table mangle -A OUTPUT  -d 173.242.119.141 -j LOG --log-prefix "output end"
root@OpenWrt:~# ping -I wana -c 5 139.59.209.225
PING 139.59.209.225 (139.59.209.225): 56 data bytes
64 bytes from 139.59.209.225: seq=0 ttl=52 time=556.206 ms

--- 139.59.209.225 ping statistics ---
5 packets transmitted, 1 packets received, 80% packet loss
round-trip min/avg/max = 556.206/556.206/556.206 ms

[  249.122255] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  249.136949] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.152521] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.168472] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.333152] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  249.347822] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  249.363379] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  249.379328] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.186954] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  250.201625] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.217190] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.232704] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.186778] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  251.201465] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.217046] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.232565] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.327170] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  251.341842] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.357411] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.372925] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.475861] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  251.490533] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.506101] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.521614] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.556966] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  251.571631] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.587199] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.602711] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.618711] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  251.633392] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.648941] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.664455] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.680630] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  251.695306] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.710856] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.726370] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.742266] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  251.756962] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.772524] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.788038] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.804120] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  251.818804] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.834357] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.849872] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.866016] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  251.880694] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.896242] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.911751] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.927804] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  251.942477] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.958027] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.973537] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.989584] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  252.004261] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.019812] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.035327] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.051684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  252.066361] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.081930] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.097446] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.114117] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  252.128796] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.144346] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.159860] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.175950] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  252.190652] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.206209] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.221724] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.239147] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  252.253998] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.269732] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.285423] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.301758] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  252.316440] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.331993] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.347514] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.363793] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  252.378646] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.394367] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.410052] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.426284] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  252.440958] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.456506] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.472030] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.488500] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  252.503349] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.519074] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.534768] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.553980] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  252.568674] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.584232] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.599747] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.675091] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  252.689932] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.705673] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.721361] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.753382] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  252.768055] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.783622] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.799139] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.865110] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  252.879799] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.895370] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.910887] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.951684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  252.966358] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  252.981930] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  252.997445] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  253.038968] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  253.053649] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.069215] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.084746] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.100904] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  253.115603] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.131154] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.146669] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.162729] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  253.177408] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.192957] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.208470] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.224456] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  253.239157] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.254736] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.270239] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.287451] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  253.302141] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.317689] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.333198] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.349109] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  253.363799] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.379351] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.394863] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.410856] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  253.425530] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.441075] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.456588] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.472620] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  253.487295] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.502840] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.518358] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.534507] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  253.549184] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.564745] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.580254] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.598744] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  253.613420] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.628971] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.644481] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.660503] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  253.675177] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.690723] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.706235] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.722383] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  253.737060] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.752611] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.768124] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.784437] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  253.799290] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.815018] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.830702] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.847020] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  253.861701] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.877255] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.892768] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.908828] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  253.923677] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.939410] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.955099] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.971337] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  253.986012] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.001559] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.017071] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.033082] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  254.047933] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.063656] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.079340] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.095684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  254.110393] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.125950] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.141462] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.157710] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  254.172571] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.188303] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.203990] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.220430] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  254.235108] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.250658] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.266183] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.301667] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  254.316346] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.331919] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.347435] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.379434] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  254.394102] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.409669] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.425184] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.465103] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  254.479766] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.495332] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.510848] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.526845] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  254.541534] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.557087] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.572599] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.588587] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  254.603261] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.618811] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.634325] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.652781] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  254.667459] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.683005] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.698517] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.714503] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  254.729374] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.745106] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.760792] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.777014] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  254.791881] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.807608] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.823300] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.839564] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  254.854413] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.870135] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.885822] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.902179] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  254.917032] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.932758] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.948444] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  264.799728] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] 
[  264.823990] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  264.849151] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  264.874691] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  266.067228] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  266.081894] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  266.097450] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  266.113415] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  270.171048] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] 
[  270.195326] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  270.220481] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  270.246034] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  271.826512] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  271.841192] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  271.856764] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  271.872726] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  272.028888] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  272.043384] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00 
[  272.058765] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00 
[  272.074541] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00

root@OpenWrt:~# tcpdump -i wana -nn host 139.59.209.225 -vv -X
tcpdump: listening on wana, link-type RAW (Raw IP), capture size 262144 bytes
01:30:25.659203 IP (tos 0x0, ttl 64, id 62017, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 0, length 64
        0x0000:  4500 0054 f241 4000 4001 e03b 0a0a 0105  E..T.A@.@..;....
        0x0010:  8b3b d1e1 0800 52c4 6a1a 0000 7c9c be84  .;....R.j...|...
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:30:26.214558 IP (tos 0x0, ttl 52, id 3042, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 27162, seq 0, length 64
        0x0000:  4500 0054 0be2 0000 3401 129c 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 5ac4 6a1a 0000 7c9c be84  ......Z.j...|...
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:30:26.727234 IP (tos 0x0, ttl 64, id 62059, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 1, length 64
        0x0000:  4500 0054 f26b 4000 4001 e011 0a0a 0105  E..T.k@.@.......
        0x0010:  8b3b d1e1 0800 0977 6a1a 0001 b5e8 ce84  .;.....wj.......
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:30:27.727645 IP (tos 0x0, ttl 64, id 62064, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 2, length 64
        0x0000:  4500 0054 f270 4000 4001 e00c 0a0a 0105  E..T.p@.@.......
        0x0010:  8b3b d1e1 0800 2132 6a1a 0002 8e2c de84  .;....!2j....,..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:30:28.779427 IP (tos 0x0, ttl 64, id 62102, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 3, length 64
        0x0000:  4500 0054 f296 4000 4001 dfe6 0a0a 0105  E..T..@.@.......
        0x0010:  8b3b d1e1 0800 8624 6a1a 0003 1939 ee84  .;.....$j....9..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:30:29.827961 IP (tos 0x0, ttl 64, id 62108, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 4, length 64
        0x0000:  4500 0054 f29c 4000 4001 dfe0 0a0a 0105  E..T..@.@.......
        0x0010:  8b3b d1e1 0800 9f23 6a1a 0004 f038 fe84  .;.....#j....8..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
^C
6 packets captured
6 packets received by filter
0 packets dropped by 

root@VPS-Server:~# tcpdump -i wg0 -nn host 139.59.209.225 -vv -X
tcpdump: listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
09:30:26.010613 IP (tos 0x0, ttl 64, id 62017, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 27162, seq 0, length 64
        0x0000:  4500 0054 f241 4000 4001 e03b 0a0a 0105  E..T.A@.@..;....
        0x0010:  8b3b d1e1 0800 52c4 6a1a 0000 7c9c be84  .;....R.j...|...
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:30:26.152118 IP (tos 0x0, ttl 52, id 3042, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 27162, seq 0, length 64
        0x0000:  4500 0054 0be2 0000 3401 129c 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 5ac4 6a1a 0000 7c9c be84  ......Z.j...|...
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
^C
2 packets captured
2 packets received by filter
0 packets dropped by kernel

Mwan3 stoped:

root@OpenWrt:~# mwan3 stop
root@OpenWrt:~# ping -I wana -c 5 139.59.209.225
PING 139.59.209.225 (139.59.209.225): 56 data bytes
64 bytes from 139.59.209.225: seq=0 ttl=52 time=338.852 ms
64 bytes from 139.59.209.225: seq=1 ttl=52 time=338.575 ms
64 bytes from 139.59.209.225: seq=2 ttl=52 time=337.782 ms
64 bytes from 139.59.209.225: seq=3 ttl=52 time=338.441 ms
64 bytes from 139.59.209.225: seq=4 ttl=52 time=337.989 ms

--- 139.59.209.225 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 337.782/338.327/338.852 ms

[  437.043153] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47294 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  437.057819] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47294 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  437.072321] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47294 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  437.087216] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47294 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  437.105578] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=47298 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  437.120240] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=47298 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  437.134718] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=47298 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  437.149611] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=47298 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  437.304709] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47305 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  437.319195] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47305 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  437.333515] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47305 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  437.348238] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47305 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  438.167201] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47320 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  438.181869] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47320 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  438.196366] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47320 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  438.211274] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47320 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  439.167819] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47387 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  439.182487] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47387 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  439.196980] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47387 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  439.211876] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47387 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  440.168429] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47392 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  440.183097] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47392 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  440.197588] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47392 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  440.212486] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47392 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  441.169025] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47488 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  441.183693] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47488 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  441.198185] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47488 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  441.213081] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=47488 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  451.651639] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47570 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  451.666123] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47570 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  451.680439] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47570 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  451.695162] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=47570 PROTO=UDP SPT=53766 DPT=51888 LEN=40 

root@OpenWrt:~# tcpdump -i wana -nn host 139.59.209.225 -vv -X
tcpdump: listening on wana, link-type RAW (Raw IP), capture size 262144 bytes
01:33:33.596211 IP (tos 0x0, ttl 64, id 9799, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 0, length 64
        0x0000:  4500 0054 2647 4000 4001 ac36 0a0a 0105  E..T&G@.@..6....
        0x0010:  8b3b d1e1 0800 10fd 4825 0000 ac4d f28f  .;......H%...M..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:33.934288 IP (tos 0x0, ttl 52, id 37378, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 0, length 64
        0x0000:  4500 0054 9202 0000 3401 8c7b 8b3b d1e1  E..T....4..{.;..
        0x0010:  0a0a 0105 0000 18fd 4825 0000 ac4d f28f  ........H%...M..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:34.720375 IP (tos 0x0, ttl 64, id 9858, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 1, length 64
        0x0000:  4500 0054 2682 4000 4001 abfb 0a0a 0105  E..T&.@.@.......
        0x0010:  8b3b d1e1 0800 9fd4 4825 0001 0c75 0390  .;......H%...u..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:35.058222 IP (tos 0x0, ttl 52, id 37539, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 1, length 64
        0x0000:  4500 0054 92a3 0000 3401 8bda 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 a7d4 4825 0001 0c75 0390  ........H%...u..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:35.721077 IP (tos 0x0, ttl 64, id 9943, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 2, length 64
        0x0000:  4500 0054 26d7 4000 4001 aba6 0a0a 0105  E..T&.@.@.......
        0x0010:  8b3b d1e1 0800 928e 4825 0002 0aba 1290  .;......H%......
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:36.058134 IP (tos 0x0, ttl 52, id 37558, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 2, length 64
        0x0000:  4500 0054 92b6 0000 3401 8bc7 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 9a8e 4825 0002 0aba 1290  ........H%......
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:36.721779 IP (tos 0x0, ttl 64, id 9997, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 3, length 64
        0x0000:  4500 0054 270d 4000 4001 ab70 0a0a 0105  E..T'.@.@..p....
        0x0010:  8b3b d1e1 0800 8548 4825 0003 08ff 2190  .;.....HH%....!.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:37.059487 IP (tos 0x0, ttl 52, id 37613, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 3, length 64
        0x0000:  4500 0054 92ed 0000 3401 8b90 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 8d48 4825 0003 08ff 2190  .......HH%....!.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:37.722482 IP (tos 0x0, ttl 64, id 10056, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 4, length 64
        0x0000:  4500 0054 2748 4000 4001 ab35 0a0a 0105  E..T'H@.@..5....
        0x0010:  8b3b d1e1 0800 7702 4825 0004 0744 3190  .;....w.H%...D1.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
01:33:38.059745 IP (tos 0x0, ttl 52, id 37839, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 4, length 64
        0x0000:  4500 0054 93cf 0000 3401 8aae 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 7f02 4825 0004 0744 3190  ........H%...D1.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel

root@VPS-Server:~# tcpdump -i wg0 -nn host 139.59.209.225 -vv -X
tcpdump: listening on wg0, link-type RAW (Raw IP), capture size 262144 bytes
09:33:33.734953 IP (tos 0x0, ttl 64, id 9799, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 0, length 64
        0x0000:  4500 0054 2647 4000 4001 ac36 0a0a 0105  E..T&G@.@..6....
        0x0010:  8b3b d1e1 0800 10fd 4825 0000 ac4d f28f  .;......H%...M..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:33.876560 IP (tos 0x0, ttl 52, id 37378, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 0, length 64
        0x0000:  4500 0054 9202 0000 3401 8c7b 8b3b d1e1  E..T....4..{.;..
        0x0010:  0a0a 0105 0000 18fd 4825 0000 ac4d f28f  ........H%...M..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:34.859625 IP (tos 0x0, ttl 64, id 9858, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 1, length 64
        0x0000:  4500 0054 2682 4000 4001 abfb 0a0a 0105  E..T&.@.@.......
        0x0010:  8b3b d1e1 0800 9fd4 4825 0001 0c75 0390  .;......H%...u..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:35.000517 IP (tos 0x0, ttl 52, id 37539, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 1, length 64
        0x0000:  4500 0054 92a3 0000 3401 8bda 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 a7d4 4825 0001 0c75 0390  ........H%...u..
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:35.859984 IP (tos 0x0, ttl 64, id 9943, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 2, length 64
        0x0000:  4500 0054 26d7 4000 4001 aba6 0a0a 0105  E..T&.@.@.......
        0x0010:  8b3b d1e1 0800 928e 4825 0002 0aba 1290  .;......H%......
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:36.000348 IP (tos 0x0, ttl 52, id 37558, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 2, length 64
        0x0000:  4500 0054 92b6 0000 3401 8bc7 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 9a8e 4825 0002 0aba 1290  ........H%......
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:36.861590 IP (tos 0x0, ttl 64, id 9997, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 3, length 64
        0x0000:  4500 0054 270d 4000 4001 ab70 0a0a 0105  E..T'.@.@..p....
        0x0010:  8b3b d1e1 0800 8548 4825 0003 08ff 2190  .;.....HH%....!.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:37.002001 IP (tos 0x0, ttl 52, id 37613, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 3, length 64
        0x0000:  4500 0054 92ed 0000 3401 8b90 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 8d48 4825 0003 08ff 2190  .......HH%....!.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:37.861459 IP (tos 0x0, ttl 64, id 10056, offset 0, flags [DF], proto ICMP (1), length 84)
    10.10.1.5 > 139.59.209.225: ICMP echo request, id 18469, seq 4, length 64
        0x0000:  4500 0054 2748 4000 4001 ab35 0a0a 0105  E..T'H@.@..5....
        0x0010:  8b3b d1e1 0800 7702 4825 0004 0744 3190  .;....w.H%...D1.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
09:33:38.002213 IP (tos 0x0, ttl 52, id 37839, offset 0, flags [none], proto ICMP (1), length 84)
    139.59.209.225 > 10.10.1.5: ICMP echo reply, id 18469, seq 4, length 64
        0x0000:  4500 0054 93cf 0000 3401 8aae 8b3b d1e1  E..T....4....;..
        0x0010:  0a0a 0105 0000 7f02 4825 0004 0744 3190  ........H%...D1.
        0x0020:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0030:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0040:  0000 0000 0000 0000 0000 0000 0000 0000  ................
        0x0050:  0000 0000                                ....
^C
10 packets captured
10 packets received by filter
0 packets dropped by kernel
aaronjg commented 4 years ago

Okay - very strange that the src ip changed with mwan3 running and not. 180.112.126.158 v 180.112.50.137

Also, it seems strange that this ip is not showing up in the routing tables. Did the routing tables change between posts? Before it looked like the packets were coming from 173.242.119.141.

If your configuration is changing - it becomes difficult to diagnose because we can't cross reference with the previous data that you have posted.

wackejohn commented 4 years ago

@aaronjg

Okay - very strange that the src ip changed with mwan3 running and not. 180.112.126.158 v 180.112.50.137

My ISP provide the dynamic pppoe connection, the ip addr will change on ervery reconnect. It was my mistake that I copied the previous log.

The right log:

[  249.122255] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  249.136949] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.152521] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.168472] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=29806 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  249.333152] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  249.347822] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  249.363379] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  249.379328] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29814 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.186954] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  250.201625] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.217190] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.232704] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.186778] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  251.201465] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.217046] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.232565] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29829 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  251.327170] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  251.341842] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.357411] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.372925] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29838 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  251.475861] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  251.490533] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.506101] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.521614] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29839 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  251.556966] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  251.571631] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.587199] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.602711] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29841 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  251.618711] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  251.633392] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.648941] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.664455] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29842 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  251.680630] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  251.695306] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.710856] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.726370] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29847 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  251.742266] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  251.756962] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.772524] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.788038] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29852 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  251.804120] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  251.818804] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.834357] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.849872] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29854 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  251.866016] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  251.880694] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.896242] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.911751] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29859 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  251.927804] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  251.942477] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.958027] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.973537] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29863 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  251.989584] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  252.004261] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.019812] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.035327] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29868 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  252.051684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  252.066361] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.081930] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.097446] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29872 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  252.114117] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  252.128796] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.144346] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.159860] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29873 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  252.175950] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  252.190652] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.206209] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.221724] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29878 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  252.239147] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  252.253998] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.269732] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.285423] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29882 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  252.301758] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  252.316440] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.331993] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.347514] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29886 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  252.363793] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  252.378646] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.394367] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.410052] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29889 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  252.426284] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  252.440958] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.456506] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.472030] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29896 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  252.488500] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  252.503349] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.519074] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.534768] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29900 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  252.553980] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  252.568674] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.584232] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.599747] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29906 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  252.675091] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  252.689932] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.705673] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.721361] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29914 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  252.753382] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  252.768055] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.783622] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.799139] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29915 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  252.865110] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  252.879799] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.895370] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.910887] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29920 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  252.951684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  252.966358] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  252.981930] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  252.997445] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29924 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  253.038968] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  253.053649] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.069215] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.084746] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29929 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  253.100904] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  253.115603] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.131154] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.146669] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29935 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  253.162729] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  253.177408] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.192957] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.208470] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29938 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  253.224456] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  253.239157] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.254736] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.270239] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29943 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  253.287451] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  253.302141] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.317689] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.333198] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29945 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  253.349109] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  253.363799] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.379351] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.394863] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29950 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  253.410856] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  253.425530] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.441075] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.456588] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=29952 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  253.472620] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 
[  253.487295] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.502840] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.518358] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=220 TOS=0x00 PREC=0x00 TTL=64 ID=29957 PROTO=UDP SPT=53766 DPT=51888 LEN=200 MARK=0x3f00 
[  253.534507] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  253.549184] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.564745] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.580254] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=29960 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  253.598744] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 
[  253.613420] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.628971] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.644481] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=284 TOS=0x00 PREC=0x00 TTL=64 ID=29963 PROTO=UDP SPT=53766 DPT=51888 LEN=264 MARK=0x3f00 
[  253.660503] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  253.675177] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.690723] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.706235] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=29966 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  253.722383] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 
[  253.737060] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.752611] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.768124] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=348 TOS=0x00 PREC=0x00 TTL=64 ID=29969 PROTO=UDP SPT=53766 DPT=51888 LEN=328 MARK=0x3f00 
[  253.784437] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  253.799290] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.815018] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.830702] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=29970 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  253.847020] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 
[  253.861701] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.877255] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.892768] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=412 TOS=0x00 PREC=0x00 TTL=64 ID=29973 PROTO=UDP SPT=53766 DPT=51888 LEN=392 MARK=0x3f00 
[  253.908828] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  253.923677] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.939410] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.955099] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=29978 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  253.971337] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 
[  253.986012] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.001559] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.017071] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=476 TOS=0x00 PREC=0x00 TTL=64 ID=29981 PROTO=UDP SPT=53766 DPT=51888 LEN=456 MARK=0x3f00 
[  254.033082] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  254.047933] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.063656] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.079340] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=29985 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.095684] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 
[  254.110393] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.125950] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.141462] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=540 TOS=0x00 PREC=0x00 TTL=64 ID=29987 PROTO=UDP SPT=53766 DPT=51888 LEN=520 MARK=0x3f00 
[  254.157710] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  254.172571] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.188303] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.203990] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=29991 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.220430] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 
[  254.235108] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.250658] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.266183] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=604 TOS=0x00 PREC=0x00 TTL=64 ID=29994 PROTO=UDP SPT=53766 DPT=51888 LEN=584 MARK=0x3f00 
[  254.301667] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 
[  254.316346] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.331919] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.347435] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=668 TOS=0x00 PREC=0x00 TTL=64 ID=29996 PROTO=UDP SPT=53766 DPT=51888 LEN=648 MARK=0x3f00 
[  254.379434] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 
[  254.394102] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.409669] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.425184] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=732 TOS=0x00 PREC=0x00 TTL=64 ID=29997 PROTO=UDP SPT=53766 DPT=51888 LEN=712 MARK=0x3f00 
[  254.465103] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 
[  254.479766] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.495332] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.510848] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=796 TOS=0x00 PREC=0x00 TTL=64 ID=29998 PROTO=UDP SPT=53766 DPT=51888 LEN=776 MARK=0x3f00 
[  254.526845] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 
[  254.541534] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.557087] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.572599] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=860 TOS=0x00 PREC=0x00 TTL=64 ID=30004 PROTO=UDP SPT=53766 DPT=51888 LEN=840 MARK=0x3f00 
[  254.588587] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 
[  254.603261] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.618811] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.634325] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=924 TOS=0x00 PREC=0x00 TTL=64 ID=30010 PROTO=UDP SPT=53766 DPT=51888 LEN=904 MARK=0x3f00 
[  254.652781] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 
[  254.667459] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.683005] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.698517] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=988 TOS=0x00 PREC=0x00 TTL=64 ID=30015 PROTO=UDP SPT=53766 DPT=51888 LEN=968 MARK=0x3f00 
[  254.714503] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 
[  254.729374] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.745106] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.760792] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1052 TOS=0x00 PREC=0x00 TTL=64 ID=30018 PROTO=UDP SPT=53766 DPT=51888 LEN=1032 MARK=0x3f00 
[  254.777014] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 
[  254.791881] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.807608] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.823300] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1116 TOS=0x00 PREC=0x00 TTL=64 ID=30024 PROTO=UDP SPT=53766 DPT=51888 LEN=1096 MARK=0x3f00 
[  254.839564] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 
[  254.854413] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.870135] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.885822] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1180 TOS=0x00 PREC=0x00 TTL=64 ID=30028 PROTO=UDP SPT=53766 DPT=51888 LEN=1160 MARK=0x3f00 
[  254.902179] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 
[  254.917032] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.932758] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  254.948444] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=1244 TOS=0x00 PREC=0x00 TTL=64 ID=30030 PROTO=UDP SPT=53766 DPT=51888 LEN=1224 MARK=0x3f00 
[  264.799728] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] 
[  264.823990] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  264.849151] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  264.874691] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54429 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42189 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  266.067228] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  266.081894] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  266.097450] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  266.113415] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=30890 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  270.171048] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] 
[  270.195326] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  270.220481] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  270.246034] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=204 TOS=0x08 PREC=0xC0 TTL=64 ID=54710 PROTO=ICMP TYPE=3 CODE=3 [SRC=173.242.119.141 DST=180.112.50.137 LEN=176 TOS=0x08 PREC=0x00 TTL=54 ID=42218 PROTO=UDP SPT=51888 DPT=53766 LEN=156 ] MARK=0x3f00 
[  271.826512] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 
[  271.841192] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  271.856764] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  271.872726] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=176 TOS=0x08 PREC=0x80 TTL=64 ID=31053 PROTO=UDP SPT=53766 DPT=51888 LEN=156 MARK=0x3f00 
[  272.028888] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 
[  272.043384] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00 
[  272.058765] postroute startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00 
[  272.074541] postroute endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=31070 PROTO=UDP SPT=53766 DPT=51888 LEN=40 MARK=0x3f00
aaronjg commented 4 years ago

Got it. So now we can see that that it is for some reason trying to route the encrypted vpn traffic back through the VPN. You can see that the log with mwan3 is much longer than the log without. After the first successful ping, each ping attempt triggers a cascade of 17 packets, each 64 bytes longer than the previous.

Is your VPN perhaps connected on port 443? If so can you delete the rule titled 'https?'