Conflict with mwan3

[mfp20]

root@OpenWrt:~# iptables -t mangle -L Chain PREROUTING (policy ACCEPT) target prot opt source destination
VPR_PREROUTING all -- anywhere anywhere [goto] mark match 0x0/0xff0000 mwan3_hook all -- anywhere anywhere
...

At boot, sometimes, the VPR_PREROUTING and mwan3_hook priority is inverted. As a result the VPN rules are ignored.

Currently I don't have the second wan up&running so I can't test what happens to mwan rules but I suspect that VPR bypasses mwan. I need to use both: any idea?

Regards

[ahmar20]

I can confirm that this issue exists with mwan3. I only use mwan3 for to keep my wan interface up and running and just to see if internet is working fine. But as soon as I installed it VPR stopped working and all of the rules in config were routing through wan instead of VPN.

[mfp20]

It looks like mwan3 set and unset it's mwan3_hook rule every time one of the wans' statuses change. The result is that the mwan3_hook can be both over or under the VPR_PREROUTING rule. As a result, whether vpr or mwan3 is ignored.

To make then both work together is not trivial. It would make the chains much more complex and in any case it is not possible to ignore the marks set by other running apps. They can't keep going as isolated projects. Probably the best solution is to merge VPR functionalities into mwan3, and rename both into something more abstract and general, ex: "Policy Routing Manager".

[stangri]

Is it just the matter which chain is listed first? Can it be resolved by restarting VPR on mwan3 restart?

[mfp20]

@stangri I'd like it could. First of all you should add two triggers: "vpr restart on mwan3 restart", as you said; and "vpr restart on mwan3 (any) wan status change". The second one is needed because mwan3 resets its mwan3_hook at every wan status change; not only at mwan3 start. In this way the VPR_PREROUTING rule comes first … but this completely bypasses mwan3_hook, effectively disabling mwan3!!!

The point here is that both services are exploiting the same iptables capability to mark, match and route. And because they are not aware of each other, the resulting routing is unpredictable, ie: both fails their goal to route traffic according to user's rules. In other words: making VPR aware of mwan3 is not enough, there's also the need for mwan3 to be aware of VPR.

I try to draft a solution here.

As a start: 1) Then mwan3 should add its rule

mwan3_hook all -- anywhere anywhere

ALWAYS at the end of the PREROUTING table. It's just a catch-all rule in order to being able to route even when the user (trough VPR) didn't set up any custom rule. A kind of fallback. If I remember correctly you can specify on the command line if the rule must be added at the top, at the end and even at specific position using line numbering. This mod must be made in the mwan3 package, in order to avoid for mwan3 to take over the whole policy routing facility.

2) VPR should then set it's rule

VPR_PREROUTING all -- anywhere anywhere [goto] mark match 0x0/0xff0000

ALWAYS at the beginning.

So that the PREROUTING table is always

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
VPR_PREROUTING all -- anywhere anywhere [goto] mark match 0x0/0xff0000
mwan3_hook all -- anywhere anywhere

3) The last rule of VPR_PREROUTING table should be

mwan3_hook all -- anywhere anywhere

So that any unmatched packet/stream/session would go back to mwan3. Again, it's a fallback rule.

4) Add in VPR rules form, the chance to configure conditionals according to mwan3 current configuration. Something like: "IF wan1 is enabled&&available THEN apply rule1 ELSE IF wan1&&wan2 are enabled&&available THEN apply rule1,rule2,rule3 ...". In this way VPR is aware of mwan3. Probably the easiest way is to make VPR build a list of possible statuses, and add a single multiple-choice list field to the form, in order to let the user choose when the rule apply.

5) In order to avoid any future developer/user to bother any other, and skip the need of constant development when a new software comes into play, there should be some guide lines in the openwrt project about iptable mark/match bits. I don't have time now to dig into iptables docs but "mark match 0x0/0xff0000" tells me that there is a marking somewhere; and later on both VPR and mwan3 match the mark to take routing decisions. Just an example: let's say that 0xff0000 is the mark. Those are 24 bits. So we could reserve the less significant 8 bits for HW/OS developers, the middle byte for applications internal rules and the remaining most significant byte for user's needs/customizations. Something like that.

I think that if you make the mods described at point 2, 3 and 4; and set the two triggers as you suggested (as at the beginning of this message); we could get some good results already.

At the same time file an issue at mwan3 repo to ask for (1), and another one at openwrt repo to ask for (5).

[stangri]

@mfp20 does mwan only insert its PREROUTING chain into mangle table or does it also insert any of the FORWARD/INPUT/OUTPUT chains? Do I understand you correctly that you need VPR_PREROUTING chain parsed before mwan3's?

[stangri]

@mfp20 VPR chains should be traversed before any others in 0.0.2-031, please reopen this issue if they are not.

[stangri]

Just a heads up, this default behaviour is changing to APPEND again in 0.0.7-7. You can alter it with the iptables_rule_option option, check the README part of config options for details.

[mfp20]

Just a quick note to say that I'm still here, and still need to make this thing work. I apologize for not sparring with you @stangri , I've been busy on other issues in these months. I'll be back to this issue soon, as I need to restructure my network config soon. As soon as I can access the equipment I'll start bothering you again! Thank you for keeping this issue in your files, cheers.