mwan3: issue with wireguard and subnet routing.

[Lars]

I have been using mwan3 to failover traffic over two wans on my router for a while. But right now I've made two changes that are causing troubles, 1 is upgrading to 20.2 and 2 is switching my VPN to wireguard.

I have a wireguard setup up and working with 4 peers, one is a client the other two are dad's home to access the lan, and my VPS which I want to route some traffic through.

I have only one rule (depending on only one policy with one member) to route a certain subnet in my network through the VPN (I'm not routing all my traffic). I have a 192.168.8.0/23 network and I want to route all 192.168.9.x IPs via the VPN.

The inital setup (today) was working in 18.06, but then I decided to finally brave it and upgrade to 20.2. And it's not working.

After lots and lots of tests and tcpdump and everything I decided to do something... wireguard interfaces (aparently) do not have default gateways. I have 3 peers, but only one of the peers, my vps, has 0.0.0.0/0 on its allowed IPs, and it's the only peer whose add routes is deselected (all other peers need it selected to access their in-vpn ip or their lans. I don't want to use my VPS as a hub).

I thought the presence of this 0.0.0.0/0 would be enough, if the package is handed to the interface, but apparently not. i did the test of manualy adding a default route to the table that mwan3 configured for this member, and everything started working:

ip route add table 4 0.0.0/0 dev wireguard via 10.42.42.1

My question is, is this really necessary then? and if so, how can I add this "default gw" (I was supposing internal routing from the wg interface should just have taken over) in a way that's automated, and integrates well with mwan3 please.

[brada4]

You need to use specific route, and you claimed 'have' networks overlap.

[Lars]

When you say I need to use a specific route, you mean a default gateway like I'm manually doing now? is there an mwan3/openwrt compatible way to define this in config or a script? or a static route? What kind of route, where?

About network overlap, the peers all have one ip inside a "vpn subnet" but the permited ips on this network do not overlap, they're all defined as /32 on the peer

[brada4]

You use different subnet in each location, and each router knows to reach other subnets, no need for individual routes. Tthere is no need for default route over vpn unless you want to have all traffic via your moms

[Lars]

You use different subnet in each location, and each router knows to reach other subnets, no need for individual routes. Tthere is no need for default route over vpn unless you want to have all traffic via your moms

Yet, if I select certain traffic on mwan3 to route via the wireguard interface, it will not work unless I add a default route gateway on the corresponding routing table manually...

[brada4]

I dont know how you 'route' overlapping subnets in different sites.

[Lars]

Will you please tell me what overlapping nets you're referring to?

[brada4]

I have a 192.168.8.0/23 network and I want to route all 192.168.9.x IPs via the VPN

[Lars]

Correct, 192.168.9.* is part of the 192.168.8.0/23 subnet. They're not overlaping subnets, it's one subnet.

My problem is with mwan3 to route ANYTHING through the wireguard VPN, be it a single ip, part of a subnet, a specific protocol, a destination port, etc, ANYTHING, it won't work unless I manually add a default route to the routing table being created by mwan3 for the vpn interface.

I have no troubles with the selection part of the rule, that works great. I have troubles with routing through the VPN.

If I create a rule based on a destination port, or IP only, I get the same problem.

If I change the catch all rule to routing through the VPN, I have the same problem.

[brada4]

/23 will never route anything to half of it in other place. you need either unique IPs in different sites or bridge via gretap over ipv4 vpn

[Lars]

I think we're not understanding eachother. Let me explain my setup and problem again.

I have sites A, B, C and D. Each site has all other three as peers.

D is an client on my phone. So it has no defined ip as a peer in any of the other sites.

Site A's wg ip is 10.42.42.1, and its allowed ips in all other sites are 10.42.42.1/32 and 0.0.0.0/0.

In Site B, which I'm discussing about and has OpenWRT, the wg interface ip is 10.42.42.2 and there's the 192.168.8.0/23 LAN. Its "allowed ips" in all other sites are 10.42.42.2/32 and 192.168.8.0/23

Site C uses a mikrotic, its wg ip is 10.42.42.3 and has the lan 192.168.0.0/23 inside, its allowed ips in other peers are 10.42.42.3/32 and 192.168.0.0/23

Site D is an android phone client, its wireguard ip is 10.42.42.4 its allowed ips in all other peers is 10.42.42.4/32

On site B, the OpenWRT, peer C has the "Route Allowed IPs" enabled, all other peers do not (it is my understanding that if I enable this for site A which has 0.0.0.0/0 as an allowed IP this will cause all my traffic to route through the VPN)

Now, I can ping any allowed ip from any peer in any site, I need to initiate a ping/connection first on site D since the IPs are dynamic. The IPs are dynamic on site B too, I have a dyn-dns setup for site B which works with sites A and D, but I have to start a connection from B to C before it'll work on the inverse since Mikrotik has troubles (still) defining a peer via hostname instead of IP.

My problem is on site B. I have 2 wans in a failover configuration via mwan3.

On site B, when I try to mark any traffic to be routed via the VPN, to the wide world, this will not work unless I find the table created for the VPN's interface (by mwan3), and add a default route to it. Since I'm routing everything via site A which has the 0.0.0.0/0 in its allowed ips, when I add a default route for this to work I use its wg ip, 10.42.42.1:

ip route add table 4 dev wireguard default via 10.42.42.1

Right now, on rules I'm using one to select all traffic whose source ip is in the 192.168.9.* part of my 192.168.8.0/23 subnet. But this problem occurs no matter what type of rule I use to select traffic to route through the vpn.

These rules rely on the "wg_only" policy which only has member "wg_m1_w3" assigned, which is a member containing the "wireguard" interface with metric 1 and weight 3.

I hope my problem is a little clearer now.

On OpenWRT, while using MWAN3 for failover wan interfaces, I can only route traffic through the wireguard VPN to the wider internet, to the one and only of my peers that has 0.0.0.0/0 as an allowed ip (site a), if and only if I manually add a default route pointing to that peer's wireguard ip ad gw to the routing table created for the wireguard interface in OpenWRT by mwan3.

I would've tried this setup with the vpn policy routing package, if I hadn't had the need for the failover setup (and I once had two rules using a balanced setup, and one internal ip using my secondary wan exclusively).

[Lars]

[brada4]

Each router needs 2 specific routes attached to respective wireguard interfaces. Android could use just an address i each site's network?

[Lars]

You mean static routes to the allowed IPs on each of its peers?

All routes here are added to the routing table, with the exception of route 0.0.0.0/0 (and 10.42.42.1, same peer) on my OpenWRT router because I understand this would lead to all traffic routing through the VPN. I can still png 10.42.42.1 from the router (and the lan inside). Are you suggesting I add a static route to 10.42.42.1 as well? and to 0.0.0.0/0? if this, how can I add that second static route without causing all my traffic to go through the VPN?

[brada4]

192.168/23 via wg0

[Lars]

Can you explain a little more please? are you saying I should make a static route for my lan through my wg interface? are you saying I should make a rule on mwan3 for my whole lan for the wg interface?

Excuse me, but, are you trolling me?

[brada4]

mwan3 is just physical media in this picture, wg will take one or other path there. It does not change anything in wg-s picture. You set route to other LAN via WG and and same in other home. Rest of the 'net works via default route everywhere.

[Lars]

That's why I came to ask how to provide a default route for a table for the wireguard interface "correctly" in a way that works with/within mwan3.

[Lars]

If I let the peer with 0.0.0.0/0 allowed ips create routes (I changed the metric for the interface to a larger one than my two wans just in case), it does work out of the box, but sketchily, it starts working and then fails and then starts again. Also afetr doing this, when I can ping the ip on that peer, I can't ping the ip on the other peer, and vice versa.

[brada4]

You mean you cannot ping wg0 in other site?

[Lars]

To ping wg0 in Site A, I need to either tell wireguard to add the routes for the allowed ips in that site, or add a static route to the ip myself. The other peers (Sites) all have their allowed ips added to route automatically.

To send traffic through the wireguard vpn via Site A (the one with 0.0.0.0/0 allowed) using mwan3 I need to either enable adding routes to allowed_ips, which adds a route without a gateway ip, or add a static route, with or without a gw ip. But if I do either, eventually traffic stops passing through the connection, and I'm unable to ping the ip of wg0 in Site A.

If I don't manually or automatically add that default route, pings to Site A wg0 IP never seem to fail, but I can't route packets through the VPN with MWAN3

[Koolholio]

The way I find this to be easy is to Prerequisites: properly assign metrics to network interfaces. (Increase interface metrics by 10 for every physical hop as it goes out of your network) Set up WireGuard as an interface. Configure the interfaces in mwan, configure ONLY instances and MWAN metrics (specific to mwan routing) and weighting in members will adjust how it load balances

Key point: Set mwan rules according to source ip 192.168.9.0/0 to route through WireGuard_only

Memo:additionally, you can cause the WireGuard interface to transport through a particular wan with routes… static routes in this case are simply on dev WAN when destination ip is 10.42.42.0/0 … etc then use gateway 192.168.0.x (say 192.168.0.1)

However any packets DeSTined for 0.0.0.0/0 on lan2 interface (lan2 has 192.168.9.0/0) could route through 10.42.42.1 (WireGuard GW)

example (not specific but relevantly doesn’t need to specify 192.168.9.0/0 but does lan2…) ip route dev lan2 dst 0.0.0.0/0 via 10.42.42.1

Since the usual command explanation is ip route destination via GW (on) dev lan2

the key point about ip routing is the difference between dev wireguard (ipnet 10.42.42.0/0) and dev lan2 (ipnet 192.168.9.0/0)

Subnet note: remember 255.255.0.0 can see 255.255.255.0 but not the other way round

hope this helps to explain some networking basics

Routing info: Frr-RIPd and igmp might help to propagate routes but this could be design limited and run into network equipment and setup compatibility… for a more complicated set up, omcproxy (igmp and multicast listener ipv6)

when dealing with multiple sites it is important that other sites will usually see WAN addresses unless there is a tunnel.

[Koolholio]

It’s irrelevant what you set internal IP’s to unless propagating routing information between… infact when they’re the same if tunnelling goes wrong you would have a nightmare with conflicts

I presume the sites are over a WAN? What is the general android site wan addressing?… if you need dial in for your android network look for PPTP (port forwarding and GRE required - not all equipment and ISPs support GRE)

you would need tunnels between them if you want it to be a “private network” which some VPNs offer with site to site (WAN solution) look up wireguard site-to-site

IPSec is older site-to-site, L2TP works on Mac addressing, PPTP (point to point requires GRE protocol)

A way to avoid this nightmare with ip addressing is if you “use something like ddns and liken them to domains”