mwan3: unable to route traffic sourced from router

[mfullerca]

Maintainer: @feckert Environment: ar71xx, archer-c7-v2, Reboot (17.01.4, r3560-79f57e422d), mwan 2.0.2-1

Description: I upgraded from CC 15.05.1 and mwan3 1.6.x to LEDE 17.01.4 with mwan 2.0.2-1 and seem to have lost the ability to use mwan3 to selectively route packets from the router itself. In 1.6.x this was done with a "Self" interface, but that hack doesn't seem to work with 2.0 and IIRC wasn't supposed to be necessary.

In the course of debugging I've verified that everything is installed correctly, mwan3 diagnostics pass, mwan3 iptables rules are in the "mangle" table, etc. While my desired config is slightly more complex, I've distilled it down to the below which represents what I want: basically to route packets to a particular host (say 8.8.4.4) sourced from the router differently; otherwise half my traffic goes to wan and the other half to wan2. The below will successfully route 8.8.4.4 differently if the packets originate from my lan but not from the router itself:

config rule 'DNS'
    option dest_ip '8.8.4.4'
    option sticky '0'
    option use_policy 'wan2_wan'

config rule 'sticky_wan'
    option src_ip '0.0.0.128/0.0.0.128'
    option sticky '0'
    option use_policy 'wan_wan2'

config rule 'sticky_wan2'
    option src_ip '0.0.0.0/0.0.0.128'
    option sticky '0'
    option use_policy 'wan2_wan'

config rule 'default_rule'
    option dest_ip '0.0.0.0/0'
    option proto 'all'
    option sticky '0'
    option use_policy 'unreachable'

config interface 'wan'
    option enabled '1'
    option reliability '2'
    option count '1'
    option timeout '2'
    list track_ip '8.8.4.4'
    list track_ip '208.67.222.222'
    list track_ip '76.14.96.14'
    option interval '30'
    option down '2'
    option up '4'

config interface 'wan2'
    option enabled '1'
    option reliability '2'
    option count '1'
    option timeout '2'
    option down '2'
    option up '4'
    option interval '30'
    list track_ip '8.8.8.8'
    list track_ip '208.67.220.220'
    list track_ip '75.75.76.76'

config member 'wan_m1_w3'
    option interface 'wan'
    option metric '1'
    option weight '3'

config member 'wan_m2_w3'
    option interface 'wan'
    option metric '2'
    option weight '3'

config member 'wan_m3_w3'
    option interface 'wan'
    option metric '3'
    option weight '3'

config member 'wan2_m1_w2'
    option interface 'wan2'
    option metric '1'
    option weight '1'

config member 'wan2_m2_w2'
    option interface 'wan2'
    option metric '2'
    option weight '1'

config member 'wan2_m3_w2'
    option interface 'wan2'
    option metric '3'
    option weight '1'

config policy 'wan_only'
    list use_member 'wan_m1_w3'

config policy 'wan2_only'
    list use_member 'wan2_m1_w2'

config policy 'balanced'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m1_w2'

config policy 'wan_wan2'
    list use_member 'wan_m1_w3'
    list use_member 'wan2_m2_w2'

config policy 'wan2_wan'
    list use_member 'wan_m2_w3'
    list use_member 'wan2_m1_w2'

[feckert]

@mfullerca did you add a "config globals 'globals'" section in the mwan3 config, with the option "option local_source 'lan'"? This should fix your issue. This script is responsible for this things https://github.com/openwrt/packages/blob/master/net/mwan3/files/etc/hotplug.d/iface/14-mwan3 This will add the ip from the lan interface to the local interface (lo) and so the package which the router generates have the sourc ip of the lan interface. Router package will then be treated as they come from the lan and will so pass the mangle tables.

[mfullerca]

@feckert is this option documented somewhere? Because I thought I searched around for a solution pretty thoroughly.

Anyhow, I first tried it under mwan3-2.0.2-1 (the version released for LEDE 17.01.4) and it didn't work. I noticed that etc/hotplug.d/iface/14-mwan3 doesn't even exist for that version, so I upgraded to mwan3-2.6.7 and it still didn't work.

Some debugging revealed that etc/hotplug.d/iface/14-mwan3 was only being run for the wan interfaces and not for lan. I eventually added a dummy interface for lan in mwan3 (tracked, but no rules) so that mwan3 was aware of it, and then it did run etc/hotplug.d/iface/14-mwan3 for lan, which triggered the ensuing logic in the script, including a default via 192.168.3.1 dev lo route entry.

However, this didn't have the intended results and I lost all my external network access, so I reverted that change. I then tried the "self" dummy interface idiom from mwan3_1.6 again and that did work, so I'm now successfully running with the dummy "self" interface.

Two other notes:

1. Running # mwan3 restart /sbin/hotplug-call: /etc/hotplug.d/iface/14-mwan3: line 31: usleep: not found

usleep doesn't exist on LEDE.

2. At some point I tried downgrading to mwan3-1.6-2 since that worked under CC 15.05.1, but too much has changed since then for it to be a drop-in replacement.

[feckert]

@mfullerca

is this option documented somewhere? Because I thought I searched around for a solution pretty thoroughly.

No sorry it is not documented for now! Feel free to help me do document this on lede forum :-) . Have not enough time :-( todo this.

However, this didn't have the intended results and I lost all my external network access, so I reverted that change. I then tried the "self" dummy interface idiom from mwan3_1.6 again and that did work, so I'm now successfully running with the dummy "self" interface.

The script 14-mwan3 should have the same result as the previous "self" interface. If the option "local_source" is set to "lan" and the "lan" interface is setting up on boot or on /etc/init.d/network resart, then the default route for the "lo" should be set default via 192.168.3.1 dev lo on your example.

Thanks for

usleep doesn't exist on LEDE.

i will fix this it is a bug!

So you have a running setup with the latest version and with the old "self" dummy interface idiom?

[mfullerca]

@feckert

No sorry it is not documented for now! Feel free to help me do document this on lede forum :-) . Have not enough time :-( todo this.

If I ever get a stable, working config I'll give it a shot, but:

The script 14-mwan3 should have the same result as the previous "self" interface. If the option "local_source" is set to "lan" and the "lan" interface is setting up on boot or on /etc/init.d/network restart, then the default route for the "lo" should be set default via 192.168.3.1 dev lo on your example.

When I add the local_source option and "lan" as an interface I can see in the table as default via 192.168.3.1 dev lo but then that breaks all my wan connectivity from the lan (which I verified by running ip route del default via 192.168.3.1 dev lo).

So you have a running setup with the latest version and with the old "self" dummy interface idiom?

I thought I did but it caused some other, slightly more subtle problem and so I had to revert it: it wasn't as bad as the above where all connectivity was broken, but it broke DNS for my first listed DNS server, so dumb devices were failing and sane computers taking 2s to retry to the second DNS server (apparently my VoIP box is dumb so I had no phone for a day).

[feckert]

When I add the local_source option and "lan" as an interface I can see in the table as default via 192.168.3.1 dev lo but then that breaks all my wan connectivity from the lan (which I verified by running ip route del default via 192.168.3.1 dev lo).

This should not change your wan connectivity. This is only for the router self initiated traffic!

Do know the Problem know? Are there any changes which have to be done in mwan3?

[thiagoc]

Hi. I'm having this problem too, but when I add that local_source option it does nothing, because it doesn't have that 14-mwan3 script. I should create it? I'm using the latest version (2.0.2-1).

When I add the "self" interface it works.

root@fw:~# ls -l /etc/hotplug.d/iface/
-rw-r--r--    1 root     root           244 Oct 17 15:46 00-netstate
-rw-r--r--    1 root     root          1450 Dec 19 03:17 15-mwan3
-rw-r--r--    1 root     root          1228 Mar 15 11:20 16-mwancustom
-rwxr-xr-x    1 root     root          1095 Dec 19 03:33 16-mwancustombak
-rw-r--r--    1 root     root           336 Oct 17 15:46 20-firewall

[feckert]

@thiagoc it was added in the master not in the lede-17.01

[ptpt52]

So PR https://github.com/openwrt/packages/pull/6515 is a better solution for this preblem?

[ptpt52]

I think this can be mark as fixed.

[hcwhan]

I have the same problem, but after I manually updated PR #6515, the problem was solved. Now works very well. tks.

[feckert]

@hcwhan thanks for the feedback @mfullerca i think this is fixed so i think we could close this issue

[feckert]

@hnyman no answer from issue reported but i think this issue is fixed. I think we could close thjis issue now.