mwan3: packets are not routed properly

[misanthropos]

Maintainer: @feckert Environment: (ath79, archer C7 v2, 19.07.1)

Description:

I tried to make an exception to route packets for a specific host directly (instead via vpn) through wlan1, but that does not work for all destinations, so I tried to figure out why and this might be the cause:

iptables -S -t mangle shows:

-A mwan3_iface_in_VPN0 -i tun0 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_VPN0 -i tun0 -m mark --mark 0x0/0x3f00 -m comment --comment VPN0 -j MARK --set-xmark 0x100/0x3f00
-A mwan3_iface_in_VPN1 -i tun1 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_VPN1 -i tun1 -m mark --mark 0x0/0x3f00 -m comment --comment VPN1 -j MARK --set-xmark 0x200/0x3f00
-A mwan3_iface_in_VPN2 -i tun2 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_VPN2 -i tun2 -m mark --mark 0x0/0x3f00 -m comment --comment VPN2 -j MARK --set-xmark 0x300/0x3f00
-A mwan3_iface_in_VPN3 -i tun3 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_VPN3 -i tun3 -m mark --mark 0x0/0x3f00 -m comment --comment VPN3 -j MARK --set-xmark 0x400/0x3f00
-A mwan3_iface_in_WWAN -i wlan1 -m set --match-set mwan3_connected src -m mark --mark 0x0/0x3f00 -m comment --comment default -j MARK --set-xmark 0x3f00/0x3f00
-A mwan3_iface_in_WWAN -i wlan1 -m mark --mark 0x0/0x3f00 -m comment --comment WWAN -j MARK --set-xmark 0x500/0x3f00

So every packet is marked the same, if I am not mistaken, hence routing is based solemnly on route metrics?

[misanthropos]

Digging a bit deeper packets might be marked correctly, but the routing table (5) which is supposed to only go through the wlan1 interface looks like this:

ip route list table 5

0.0.0.0/1 via  xx.xx.xx.xx dev tun0 metric 10 
0.0.0.0/1 via xx.xx.xx.xx dev tun1 metric 20 
0.0.0.0/1 via xx.xx.xx.xx dev tun2 metric 30 
0.0.0.0/1 via xx.xx.xx.xx dev tun3 metric 40 
xx.xx.xx.xx/26 dev tun3 proto kernel scope link src xx.xx.xx.xx
172.20.1.0/24 dev br-lan proto kernel scope link src 172.20.1.1 
xx.xx.xx.xx/26 dev tun1 proto kernel scope link src xx.xx.xx.xx
xx.xx.xx.xx/26 dev tun2 proto kernel scope link src xx.xx.xx.xx
192.168.11.0/24 via 172.20.1.10 dev br-lan proto static 
192.168.178.0/24 dev wlan1 proto static scope link metric 5 
xx.xx.xx.xx/25 dev tun0 proto kernel scope link src xx.xx.xx.xx

[misanthropos]

I think I solved the problem by setting the source address in VPN static routes to the routers internal IP.

[misanthropos]

Nope - the routing table 5 is still not setup correctly - there should be not a single vpn entry there, right? (wwan_only)

I supplied my config here: mwan3.txt

[misanthropos]

Interestingly domains marked with ipset are routed correctly. Could the problem lie in NAT?

[misanthropos]

After some more testing I can state there is something wrong, and I believe its not my config, but can not pinpoint to where the problem lies. I have created a rule with a specific destination address which should go through wwan_only and still gets routed through vpn.

mwan3.txt

[feckert]

You have to set for each network config that is tracked by mwan3 a metric in your network configuration. For example

config interface wan
option metric '10'

[misanthropos]

I did that: and every route has its own metric

the only case exceptions which work is with ipset rules coming from dnsmasq, which can be setup with vpnbypass - but to be sure its not vpnbypass messing with marking packets I disabled vpnbypass.

route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         xx.xx.xx.xx 128.0.0.0       UG    10     0        0 tun0
default         xx.xx.xx.xx 128.0.0.0       UG    20     0        0 tun1
default         xx.xx.xx.xx  128.0.0.0      UG    30     0        0 tun2
default         xx.xx.xx.xx  128.0.0.0      UG    40     0        0 tun3
default         fb7560         0.0.0.0         UG    5      0        0 wlan1
xx.xx.xx.xx   *               255.255.255.192 U     0      0        0 tun3
172.20.1.0      *               255.255.255.0   U     0      0        0 br-lan
xx.xx.xx.xx   *               255.255.255.192 U     0      0        0 tun1
xx.xx.xx.xx  *               255.255.255.192 U     0      0        0 tun2
192.168.11.0    atomix.home.lan 255.255.255.0   UG    0      0        0 br-lan
192.168.178.0   *               255.255.255.0   U     5      0        0 wlan1
xx.xx.xx.xx *               255.255.255.128 U     0      0        0 tun0

[misanthropos]

Still struggling with getting it to work properly. Some routes are going consistently through VPN, while others do not....

[feckert]

Sorry I have no setup for your problem! I am not using mwan3 tracking for VPN interfaces. Please ask in the openwrt forum if some one could help you.

[ptpt52]

@misanthropos show your /etc/config/network config

[misanthropos]

@ptpt52 https://pastebin.com/pYmMEMmf

There is nothing special in there, really. You will find some static routes which are dummies to satisfy mwan3. Same for VPN interfaces.

[ptpt52]

@misanthropos it is your setup problem.

0.0.0.0/1 via  xx.xx.xx.xx dev tun0 metric 10 
0.0.0.0/1 via xx.xx.xx.xx dev tun1 metric 20 
0.0.0.0/1 via xx.xx.xx.xx dev tun2 metric 30 
0.0.0.0/1 via xx.xx.xx.xx dev tun3 metric 40 

You better tell me your network needs, I might be able to tell you how to configure

[misanthropos]

@ptpt52

Those routes are set because mwan3 would complain otherwise and mwan3 explicitly states wan connections have to have distinct metrics. The actual metric is set by OpenVPN, when starting the connection (and are distinct too) - and it works and traffic is routed through VPN.

What I want however is to exempt specific hosts to go through VPN and get a direct WAN connection, and THAT is not working correctly.

See the configuration of the host: lake in https://github.com/openwrt/packages/files/4266734/mwan3.txt

config rule 'lake'
    option src_ip '172.20.1.22'
    option proto 'all'
    option sticky '0'
    option use_policy 'wwan_only'

All connections of that host should go through WAN! While many connections go through WAN, others still go through VPN.

[ptpt52]

0.0.0.0/1 via  xx.xx.xx.xx dev tun0 metric 10

this route is not correct for your case.

show your main routes

ip route show

[ptpt52]

@ptpt52

Those routes are set because mwan3 would complain otherwise and mwan3 explicitly states wan connections have to have distinct metrics. The actual metric is set by OpenVPN, when starting the connection (and are distinct too) - and it works and traffic is routed through VPN.

What I want however is to exempt specific hosts to go through VPN and get a direct WAN connection, and THAT is not working correctly.

See the configuration of the host: lake in https://github.com/openwrt/packages/files/4266734/mwan3.txt

config rule 'lake'
  option src_ip '172.20.1.22'
  option proto 'all'
  option sticky '0'
  option use_policy 'wwan_only'

All connections of that host should go through WAN! While many connections go through WAN, others still go through VPN.

this workaround is the problem

config route
    option target '0.0.0.0/1'
    option metric '10'
    option interface 'VPN0'

config route
    option target '0.0.0.0/1'
    option metric '20'
    option interface 'VPN1'

config route
    option target '0.0.0.0/1'
    option metric '30'
    option interface 'VPN2'

config route
    option target '0.0.0.0/1'
    option metric '40'
    option interface 'VPN3'

drop these route and should add metric to the default route for every vpn tun

[misanthropos]

your suggestion directly contradicts prepare_and_the_check_the_default_routing_table_for_the_wan_interfaces

and issuecomment-592527021

[feckert]

I do not think so this is contradicts. You have to set for each tun interface in the /etc/config/network section a different metric.

 config interface 'VPN0'
         option proto 'none'
         option ifname 'tun0'
         option metric '<value>'

[misanthropos]

check my routing table: #issuecomment-592535929

[misanthropos]

@feckert @ptpt52

So: I did what you suggested and removed the static routes.

the routing table did not change:

default xx.xx.xx.xx 128.0.0.0 UG 10 0 0 tun1 default xx.xx.xx.xx 128.0.0.0 UG 20 0 0 tun2 default xx.xx.xx.xx 128.0.0.0 UG 30 0 0 tun0 default xx.xx.xx.xx 128.0.0.0 UG 40 0 0 tun3 default fb7560.home.lan 0.0.0.0 UG 5 0 0 wlan1 xx.xx.xx.xx * 255.255.255.192 U 0 0 0 tun3 ...

Same result: some routes are going through WAN others through VPN To be clear: This is for the host which should not go through VPN at all.

[misanthropos]

Interestingly this is consistent for the destinations... eg. traceroute www.taz.de always goes through WAN traceroute www.faz.de always goes through VPN

it looks like almost all of connections still go through VPN - only some go through WAN (maybe because they match ip range from ipset?)

I believe in the end its how the mangled packets are handled (if at all) - either markings are incorrect or not handled correctly.

[feckert]

Please try the newest version 2.8.12 in master and openwrt-19.07(stable) branch. There have been some improvements.

If your problem still exist please reopen the issue.

[misanthropos]

Same behaviour as before. How can I reopen this issue?

[aaronjg]

Why do you have your routes set up as 0.0.0.0/1 Rather than a default route or 0.0.0.0/0?

This could explain why some routes are going over one interface but not the other.:

# resolveip -4 taz.de
193.104.220.21
# resolveip -4 faz.de
40.118.6.229

Can you post the output of the mwan3 troubleshooting page cgi-bin/luci/admin/status/mwan/troubleshooting?

[ptpt52]

You have to make it clear how mwan3 work, I think the problem is your wrong setup.

I can't clearly understand your real needs, but I can be sure that there is no problem with mwan3

[misanthropos]

Why do you have your routes set up as 0.0.0.0/1 Rather than a default route or 0.0.0.0/0?

That was the route my VPN provider would set if I don`t specify otherwise. I forced it to default route to make sure its not related to that and it solved the problem ^^. I am dead certain I tried the default route before and it would screw up the VPN or something else. However it seems to work now!

I am sorry for the confusion I may have stirred up - its clearly not related to mwan3.

This could explain why some routes are going over one interface but not the other.:

# resolveip -4 taz.de
193.104.220.21
# resolveip -4 faz.de
40.118.6.229

yeah... 193 translates to 11000001 so the first bit would have caused the different routing behaviour with 0.0.0.0/1

Thanks for that!