search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.01k stars 3.48k forks source link

Mwan3track not working properly on wireguard interface #10712

Closed wackejohn closed 4 years ago

wackejohn commented 4 years ago

Maintainer: @feckert Environment: OpenWrt SNAPSHOT, r11625-a512123a4b Mwan3 version: 2.8.2-2 Description: I'm using mwan3 and wireguard as client with latest openwrt snapshot. And it seemed that the issue was caused by mwan3 itself, when mwan3 was running ,the ping -I $Device not working, and then mwan3 would always mark the wireguard interface down, but the wireguard interface actually working... Below is the test resault:

root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5233ms

root@HOME-Router:~# mwan3 stop
root@HOME-Router:~# mwan3 status
Interface status:
 interface wan1 is offline and tracking is down
 interface wan1_6 is offline and tracking is down
 interface wan2 is offline and tracking is down
 interface wan2_6 is offline and tracking is down
 interface wan3 is offline and tracking is down
 interface wan3_6 is offline and tracking is down
 interface wan4 is offline and tracking is down
 interface wan4_6 is offline and tracking is down
 interface wana is offline and tracking is down
 interface wana_6 is offline and tracking is down
 interface wanb is offline and tracking is down
 interface wanb_6 is offline and tracking is down

Current ipv4 policies:

Current ipv6 policies:

Directly connected ipv4 networks:

Directly connected ipv6 networks:

Active ipv4 user rules:

Active ipv6 user rules:

root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=58 time=135 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=58 time=136 ms
^C
--- 8.8.8.8 ping statistics ---
6 packets transmitted, 5 received, 16.6667% packet loss, time 5007ms
rtt min/avg/max/mdev = 134.652/135.242/136.296/0.592 ms
root@HOME-Router:~# mwan3 restart
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
uci: Entry not found
uci: Entry not found
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
uci: Entry not found
uci: Entry not found
root@HOME-Router:~# /usr/bin/ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8) from 192.168.120.2 wana: 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7247ms

And there is a discuss on the forum: https://forum.openwrt.org/t/mwan3track-not-working-properly-on-wireguard/49557

wackejohn commented 4 years ago

@aaronjg

Is your VPN perhaps connected on port 443? If so can you delete the rule titled 'https?'

Nope. My VPN was configred on port 51888 udp, as the log showed, while the https is tcp port.

aaronjg commented 4 years ago

Many VPNs use port 443 (https port) to avoid being blocked by ISP firewalls. Is any part of your connection going over 443 or https? You included it in the minimal config - so I was wondering what it is needed for.

Something still seems wrong from the IP tables log:

SRC:180.112.50.137 DEST: 173.242.119.141

From the routing table:

117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 
173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static

Do you have any idea why the packets are getting assigned a source address of 180.112.50.137? From your routing table, it looks like the src for ppoe-wan should be 117.85.170.4

Another thing that looks weird is your mwan3 config has both a wan and a wana device, but the wana device is not part of any used policy. Can you confirm that this is in fact the minimal configuration to reproduce the bug?

wackejohn commented 4 years ago

@aaronjg

Many VPNs use port 443 (https port) to avoid being blocked by ISP firewalls. Is any part of your connection going over 443 or https?

I'm using the wireguard vpn, and when I use other vpns with tun device, the ping -I actually working, so I open this issue.

You included it in the minimal config - so I was wondering what it is needed for.

The https rule was generated by the mwan3, so it was one of the default rule of mwan3.

117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static

Because I capture the routing table and the log at the different time, so the ip addr was different, but this shouldn't matter, please ignore the different ip addr.

Another thing that looks weird is your mwan3 config has both a wan and a wana device, but the wana device is not part of any used policy. Can you confirm that this is in fact the minimal configuration to reproduce the bug?

Yes, with the default rule of mwan3, the ping -I won't working until stop the mwan3 (or use the 4.14 kernel and below).

aaronjg commented 4 years ago

Because I capture the routing table and the log at the different time, so the ip addr was different, but this shouldn't matter, please ignore the different ip addr.

When things change like this, it makes it very difficult to help. To identify the issue, we have to look for anomolies and these changes look very much like something when wrong in mwan3.

Yes, with the default rule of mwan3, the ping -I won't working until stop the mwan3 (or use the 4.14 kernel and below). A "minimal configuration" is the smallest possible configuration which gives the problematic behavior. To get a minimal configuration, you need to delete sections of the configuration until you can no longer reproduce the bug. (Either things start working again, or it breaks because some key configuration is missing). Some of these sections appear extraneous (eg policies that are not part of any rule). Which is why I asked if it was the minimal configuration.

Anyway, since you have added back the direct link route, and confirmed that the IP address from the log matches the correct outgoing IP address of your ppoe device - it appears that mwan3 is doing everything correctly. Unfortunately, this means that you have found a bug in wireguard or the kernel routing.

You can actually see it here:

[  250.186954] output startIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 
[  250.201625] output endIN= OUT=pppoe-wan SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.217190] postroute startIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00 
[  250.232704] postroute endIN= OUT=wana SRC=180.112.50.137 DST=173.242.119.141 LEN=156 TOS=0x00 PREC=0x00 TTL=64 ID=29827 PROTO=UDP SPT=53766 DPT=51888 LEN=136 MARK=0x3f00

So this packet gets marked correctly by iptables with 0x3f0. You never shared the output of ip rule list from the smaller (minimal?) configuration. But assuming nothing crazy is going on here, that firewall mark should skip the tables mwan3 added and match the main routing table.

The main routing table is (with 117.85.170.4 replaced with 180.112.50.137 presumably, and the corresponding gateway also replaced.):

default via 117.85.168.1 dev pppoe-wan proto static 
10.10.1.0/24 dev wana proto static scope link metric 10 
117.85.168.1 dev pppoe-wan proto kernel scope link src 117.85.170.4 
173.242.119.141 via 117.85.168.1 dev pppoe-wan proto static 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1

So that packet should match the fourth rule in the table and be routed out of ppoe. Somehow it appears to not matching that rule and being routed out of wana - which causes it to go back through ppoe.

So it appears that this is an issue upstream, either with wireguard or the kernel. You should try to create a minimal configuration without mwan3 to reproduce this, and post a bug report on the main openwrt project. A minimal configuration will likely contain:

If you can't get it with those rules, you may need a few more commands to reproduce the behavior - but try to recreate the behavior with as few commands as possible since that will make it easier for the openwrt developers to help out.

ptpt52 commented 4 years ago

@wackejohn hi in ip route why no default route for wana? this is not correct so in ip route list table 9

default dev wana scope link metric 30  <<== This line is not correct

10.10.1.0/24 dev wana proto static scope link metric 30 
31.40.214.127 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
112.1.96.1 dev pppoe-wan3 proto kernel scope link src 112.1.96.195 
112.3.84.1 dev pppoe-wan4 proto kernel scope link src 112.3.87.193 
114.228.153.1 dev pppoe-wan2 proto kernel scope link src 114.228.153.129 
173.242.119.141 via 180.115.231.1 dev pppoe-wan1 proto static metric 10 
180.115.231.1 dev pppoe-wan1 proto kernel scope link src 180.115.231.180 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110

It should be like: default via 10.10.1.xx dev wana proto static metric 30 you need to add it to main table and restart mwan3 and mwan3rtmon should copy it to table 9

and if you don't want all traffic go out via wana, you could set the metric large number

aaronjg commented 4 years ago

you need to add it to main table and restart mwan3 and mwan3rtmon should copy it to table 9

mwan3 does not copy over the default routes, it just inserts its own.

and if you don't want all traffic go out via wana, you could set the metric large number

With only one default route, changing the metric on the default route will not change the traffic routing

wackejohn commented 4 years ago

Thanks @aaronjg, with this pr and add default route of wireguard interface to main routing table ,this issue is fixed now.

ptpt52 commented 4 years ago

ping -I should work, so it is bug in wireguard @wackejohn
Please try test my new fix for it https://github.com/x-wrt/x-wrt/blob/master/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.patch

wackejohn commented 4 years ago

@ptpt52 Tried, but still have bugs working with two or more wireguard links.

The ttl was different from wana (121) to wanb (116):

root@HOME-Router_CZ:/etc/config# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=135.858 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=136.189 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.097 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=135.867 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=137.416 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=137.491 ms
64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.676 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=135.557 ms
64 bytes from 8.8.8.8: seq=8 ttl=116 time=135.830 ms
64 bytes from 8.8.8.8: seq=9 ttl=116 time=139.063 ms
64 bytes from 8.8.8.8: seq=10 ttl=116 time=135.592 ms
64 bytes from 8.8.8.8: seq=11 ttl=116 time=135.746 ms
^C
--- 8.8.8.8 ping statistics ---
12 packets transmitted, 12 packets received, 0% packet loss
round-trip min/avg/max = 135.557/136.365/139.063 ms
root@HOME-Router_CZ:/etc/config# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=133.802 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=136.936 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.612 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=138.964 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=136.239 ms
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 133.802/136.510/138.964 ms
root@HOME-Router_CZ:/etc/config# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=133.633 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=136.060 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=135.999 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=136.289 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=135.918 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=135.588 ms
64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.867 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=136.036 ms
64 bytes from 8.8.8.8: seq=8 ttl=116 time=135.913 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max = 133.633/135.700/136.289 ms
root@HOME-Router_CZ:/etc/config# mwan3 ifup wana
root@HOME-Router_CZ:/etc/config# ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=136.261 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=137.569 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=135.139 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=133.616 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=134.030 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=133.919 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=134.292 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=133.984 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=139.795 ms
64 bytes from 8.8.8.8: seq=9 ttl=121 time=133.475 ms
64 bytes from 8.8.8.8: seq=10 ttl=121 time=135.308 ms
64 bytes from 8.8.8.8: seq=11 ttl=121 time=135.950 ms
64 bytes from 8.8.8.8: seq=12 ttl=121 time=134.733 ms
64 bytes from 8.8.8.8: seq=13 ttl=121 time=137.229 ms
64 bytes from 8.8.8.8: seq=14 ttl=121 time=134.602 ms
64 bytes from 8.8.8.8: seq=15 ttl=121 time=139.457 ms
64 bytes from 8.8.8.8: seq=16 ttl=121 time=133.962 ms
64 bytes from 8.8.8.8: seq=17 ttl=121 time=137.164 ms
64 bytes from 8.8.8.8: seq=18 ttl=121 time=135.839 ms
^C
--- 8.8.8.8 ping statistics ---
19 packets transmitted, 19 packets received, 0% packet loss
round-trip min/avg/max = 133.475/135.596/139.795 ms
root@HOME-Router_CZ:/etc/config# ifdown wanb
root@HOME-Router_CZ:/etc/config# ping -I wanb 8.8.8.8
ping: bad address 'wanb'
root@HOME-Router_CZ:/etc/config# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=134.590 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=134.660 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=133.850 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=140.986 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=135.351 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=133.815 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=141.610 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=133.842 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=133.694 ms
64 bytes from 8.8.8.8: seq=9 ttl=121 time=133.862 ms
^C
--- 8.8.8.8 ping statistics ---
10 packets transmitted, 10 packets received, 0% packet loss
round-trip min/avg/max = 133.694/135.626/141.610 ms
root@HOME-Router_CZ:/etc/config# ifup wanb
root@HOME-Router_CZ:/etc/config# ifdown wana
root@HOME-Router_CZ:/etc/config# ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=136.513 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=137.159 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.090 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=137.653 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=136.160 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=136.246 ms
64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.954 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=136.978 ms
64 bytes from 8.8.8.8: seq=8 ttl=116 time=136.383 ms
64 bytes from 8.8.8.8: seq=9 ttl=116 time=135.880 ms
64 bytes from 8.8.8.8: seq=10 ttl=116 time=136.031 ms
^C
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 11 packets received, 0% packet loss
round-trip min/avg/max = 135.880/136.458/137.653 ms
ptpt52 commented 4 years ago

@wackejohn ttl is the issue? Seems to be normal, in a real network environment

wackejohn commented 4 years ago

@ptpt52 I think the ping packets was sent to wanb when ping -I wana (cause the ttl was changed from the secound ping, but when I ifdown wana or wanb manually, the ttl was normal).

ptpt52 commented 4 years ago

@wackejohn try this change and test again

diff --git a/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.patch b/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.
index c5ca454709..9cfb31194e 100644
--- a/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.patch
+++ b/package/network/services/wireguard/patches/100-skb-sock-sync-sk_bound_dev_if.patch
@@ -10,7 +10,7 @@ index c33e2c8..6c7476e 100644
 +              /* Can't call skb->destructor here in irq
 +               * so we just hack sync sk_bound_dev_if
 +               */
-+              skb->sk->sk_bound_dev_if = sock->sk_bound_dev_if;
++              skb->sk->sk_bound_dev_if = 0;
 +      }
        udp_tunnel_xmit_skb(rt, sock, skb, fl.saddr, fl.daddr, ds,
                            ip4_dst_hoplimit(&rt->dst), 0, fl.fl4_sport,
@@ -23,7 +23,7 @@ index c33e2c8..6c7476e 100644
 +              /* Can't call skb->destructor here in irq
 +               * so we just hack sync sk_bound_dev_if
 +               */
-+              skb->sk->sk_bound_dev_if = sock->sk_bound_dev_if;
++              skb->sk->sk_bound_dev_if = 0;
 +      }
        udp_tunnel6_xmit_skb(dst, sock, skb, skb->dev, &fl.saddr, &fl.daddr, ds,
                             ip6_dst_hoplimit(dst), 0, fl.fl6_sport,
wackejohn commented 4 years ago

@ptpt52 It's working.

root@HOME-Router_CZ:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=147.073 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=136.372 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=135.890 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=135.242 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=135.578 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=141.975 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=136.291 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=135.744 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=137.404 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max = 135.242/137.952/147.073 ms
root@HOME-Router_CZ:~# ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=137.485 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=137.247 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=136.739 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=137.696 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=136.210 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=136.195 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=135.757 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=135.749 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=139.680 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max = 135.749/136.973/139.680 ms
ptpt52 commented 4 years ago

Please do more test, I notice one line ttl=116 in ping -I wanb

ptpt52 commented 4 years ago

can you show your network?

ip route
ptpt52 commented 4 years ago

you could mask the ip

wackejohn commented 4 years ago 
root@HOME-Router_CZ:~# ip route
default via 114.228.*.* dev pppoe-wan1 proto static metric 10 
default via 222.188.*.* dev pppoe-wan2 proto static metric 15 
default via 223.66.*.* dev pppoe-wan3 proto static metric 20 
default via 112.1.*.1* dev pppoe-wan4 proto static metric 25 
default via 10.10.1.1 dev wana proto static metric 30 
default via 192.168.150.1 dev wanb proto static metric 40 
10.10.1.0/24 dev wana proto static scope link metric 30 
112.1.*.* dev pppoe-wan4 proto kernel scope link src 112.1.*.* 
114.228.*.* dev pppoe-wan1 proto kernel scope link src 114.228.*.* 
192.168.1.0/24 dev br-lan proto kernel scope link src 192.168.1.1 
192.168.2.0/24 dev vpn1 proto static scope link metric 110 
192.168.3.0/24 dev vpn1 proto static scope link metric 110 
192.168.150.0/24 dev wanb proto static scope link metric 40 
192.168.195.0/24 dev vpn0 proto static scope link metric 100 
192.168.195.2 dev vpn0 proto static scope link metric 100 
192.168.195.3 dev vpn0 proto static scope link metric 100 
192.168.195.4 dev vpn0 proto static scope link metric 100 
192.168.195.5 dev vpn0 proto static scope link metric 100 
192.168.195.6 dev vpn0 proto static scope link metric 100 
192.168.196.0/24 dev vpn1 proto static scope link metric 110 
192.168.196.2 dev vpn1 proto static scope link metric 110 
192.168.196.3 dev vpn1 proto static scope link metric 110 
222.188.*.* dev pppoe-wan2 proto kernel scope link src 222.188.*.*
223.66.*.* dev pppoe-wan3 proto kernel scope link src 223.66.*.*
ptpt52 commented 4 years ago

update: the previous fix is not correct

I post another fix here: https://github.com/x-wrt/x-wrt/blob/master/package/network/services/wireguard/patches/100-scrub-skb-sk-before-send-out.patch

wackejohn commented 4 years ago

@ptpt52 Working.

root@HOME-Router_CZ:~# ping -I wana 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=121 time=135.233 ms
64 bytes from 8.8.8.8: seq=1 ttl=121 time=135.210 ms
64 bytes from 8.8.8.8: seq=2 ttl=121 time=134.916 ms
64 bytes from 8.8.8.8: seq=3 ttl=121 time=135.305 ms
64 bytes from 8.8.8.8: seq=4 ttl=121 time=135.863 ms
64 bytes from 8.8.8.8: seq=5 ttl=121 time=135.562 ms
64 bytes from 8.8.8.8: seq=6 ttl=121 time=134.961 ms
64 bytes from 8.8.8.8: seq=7 ttl=121 time=135.389 ms
64 bytes from 8.8.8.8: seq=8 ttl=121 time=135.052 ms
^C
--- 8.8.8.8 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max = 134.916/135.276/135.863 ms
root@HOME-Router_CZ:~# ping -I wanb 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: seq=0 ttl=116 time=136.593 ms
64 bytes from 8.8.8.8: seq=1 ttl=116 time=135.974 ms
64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.143 ms
64 bytes from 8.8.8.8: seq=3 ttl=116 time=136.279 ms
64 bytes from 8.8.8.8: seq=4 ttl=116 time=136.264 ms
64 bytes from 8.8.8.8: seq=5 ttl=116 time=136.204 ms
64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.992 ms
64 bytes from 8.8.8.8: seq=7 ttl=116 time=136.520 ms
64 bytes from 8.8.8.8: seq=8 ttl=116 time=136.105 ms
64 bytes from 8.8.8.8: seq=9 ttl=116 time=136.008 ms
64 bytes from 8.8.8.8: seq=10 ttl=116 time=135.789 ms
^C
--- 8.8.8.8 ping statistics ---
11 packets transmitted, 11 packets received, 0% packet loss
round-trip min/avg/max = 135.789/136.170/136.593 ms
ptpt52 commented 4 years ago

yes that is what it should be

wackejohn notifications@github.com 于 2020年10月27日周二 08:49写道：

@ptpt52 https://github.com/ptpt52 Working.

root@HOME-Router_CZ:~# ping -I wana 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=121 time=135.233 ms 64 bytes from 8.8.8.8: seq=1 ttl=121 time=135.210 ms 64 bytes from 8.8.8.8: seq=2 ttl=121 time=134.916 ms 64 bytes from 8.8.8.8: seq=3 ttl=121 time=135.305 ms 64 bytes from 8.8.8.8: seq=4 ttl=121 time=135.863 ms 64 bytes from 8.8.8.8: seq=5 ttl=121 time=135.562 ms 64 bytes from 8.8.8.8: seq=6 ttl=121 time=134.961 ms 64 bytes from 8.8.8.8: seq=7 ttl=121 time=135.389 ms 64 bytes from 8.8.8.8: seq=8 ttl=121 time=135.052 ms ^C --- 8.8.8.8 ping statistics --- 9 packets transmitted, 9 packets received, 0% packet loss round-trip min/avg/max = 134.916/135.276/135.863 ms root@HOME-Router_CZ:~# ping -I wanb 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: seq=0 ttl=116 time=136.593 ms 64 bytes from 8.8.8.8: seq=1 ttl=116 time=135.974 ms 64 bytes from 8.8.8.8: seq=2 ttl=116 time=136.143 ms 64 bytes from 8.8.8.8: seq=3 ttl=116 time=136.279 ms 64 bytes from 8.8.8.8: seq=4 ttl=116 time=136.264 ms 64 bytes from 8.8.8.8: seq=5 ttl=116 time=136.204 ms 64 bytes from 8.8.8.8: seq=6 ttl=116 time=135.992 ms 64 bytes from 8.8.8.8: seq=7 ttl=116 time=136.520 ms 64 bytes from 8.8.8.8: seq=8 ttl=116 time=136.105 ms 64 bytes from 8.8.8.8: seq=9 ttl=116 time=136.008 ms 64 bytes from 8.8.8.8: seq=10 ttl=116 time=135.789 ms ^C --- 8.8.8.8 ping statistics --- 11 packets transmitted, 11 packets received, 0% packet loss round-trip min/avg/max = 135.789/136.170/136.593 ms

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/openwrt/packages/issues/10712#issuecomment-716906296, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABIVTYVKS6N6ZRXWRA7XP5TSMYKKVANCNFSM4JXG4K5Q .

feckert commented 4 years ago

@ptpt52 This is a wireguard problem and has nothing to do with the mwan3? Can you please send it upstream, so that it gets fixed in the wireguard.