opnsense / core

OPNsense GUI, API and systems backend
https://opnsense.org/
BSD 2-Clause "Simplified" License
3.06k stars 699 forks source link

Wireguard: No routes are configured if AllowedIPs are outside of the wireguard interface subnet #7355

Open interj opened 2 months ago

interj commented 2 months ago

Describe the bug

Hello, I think I have been affected by the reverse part of https://github.com/opnsense/core/issues/7304 I'm creating this issue because I'm not sure it can be solved in a scope of the other one - my AllowedIPs have proper netmasks configured but the routes aren't being created anyway.

I have two LANs connected with wireguard, one of them has a gateway running OPNsense. LAN#1 is 192.168.1.0/24, with gateway configured to forward some traffic to a wireguard peer 192.168.7.1 - this is the OPNsense vpn peer. OPNsense has a correct firewall configuration and a wireguard interface 192.168.7.1/25 that is reachable by the other gateway from 192.168.7.2 wireguard address. The peer from LAN#1 has allowed IPs 192.168.1.0/24, 192.168.7.2/32 configured in the OPNsense WebUI. Traffic coming from 192.168.7.2 is properly making the roundtrip, but the traffic from 192.168.1.0/24 reaches OPNsense and gets dropped without reply, because there is no route configured from 192.168.7.1 to 192.168.1.0/24.

Before one of these two upgrades (unfortunately I can't revert and compare at the moment) routes have been automatically configured:

Notice  pkg-static  opnsense upgraded: 24.1.2_1 -> 24.1.3_1 
Notice  pkg-static  opnsense upgraded: 23.7.12_5 -> 24.1.2_1

At the moment I can mitigate the issue either by NATing the traffic from LAN#1 as a 192.168.7.2 address, or by manually adding 192.168.7.2 as a gateway and configuring static route to 192.168.1.0/24 through it in the OPNsense config. This also seems to be the standard behaviour of wg-quick that will setup the needed routes for all AllowedIPs, even if they are on the network separate from the wireguard interface itself.

Expected behavior I expect the routes for all allowed IPs to be added automatically after I have configured the peer in WebUI. Alternatively a Note on AllowedIPs in WebUI or a warning when setting up allowedIPs outside of the wireguard interface subnet would be immensely helpful

Relevant log files

Before adding a static route:

⚡interj:~ € sudo tcpdump -i wg2 -t udp -vvv                                                                                                                                                                                                                                                                     [18:59:41]
tcpdump: listening on wg2, link-type NULL (BSD loopback), capture size 262144 bytes
IP (tos 0x0, ttl 63, id 1869, offset 0, flags [DF], proto UDP (17), length 79)
    192.168.1.201.43725 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 2443, offset 0, flags [DF], proto UDP (17), length 77)
    192.168.1.201.59805 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 2444, offset 0, flags [DF], proto UDP (17), length 79)
    192.168.1.201.33039 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 2445, offset 0, flags [DF], proto UDP (17), length 78)
    192.168.1.201.34644 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 3940, offset 0, flags [DF], proto UDP (17), length 81)
    192.168.1.201.58956 > 192.168.7.1.domain: [udp sum ok] ...

After adding a static route: image

⚡interj:~ € sudo tcpdump -i wg2 -t udp -vvv                                                                                                                                                                                                                                                                     
tcpdump: listening on wg2, link-type NULL (BSD loopback), capture size 262144 bytes
IP (tos 0x0, ttl 63, id 64805, offset 0, flags [DF], proto UDP (17), length 85)
    192.168.1.201.40920 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 64806, offset 0, flags [DF], proto UDP (17), length 85)
    192.168.1.201.40523 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 64, id 53882, offset 0, flags [none], proto UDP (17), length 328)
    192.168.7.1.domain > 192.168.1.201.40920: [udp sum ok] ...
IP (tos 0x0, ttl 64, id 23578, offset 0, flags [none], proto UDP (17), length 146)
    192.168.7.1.domain > 192.168.1.201.40523: [udp sum ok] ...
IP (tos 0x0, ttl 63, id 64891, offset 0, flags [DF], proto UDP (17), length 81)
    192.168.1.201.36117 > 192.168.7.1.domain: [udp sum ok] ...
IP (tos 0x0, ttl 64, id 14101, offset 0, flags [none], proto UDP (17), length 155)
    192.168.7.1.domain > 192.168.1.201.36117: [udp sum ok] ...

Environment

Software version used and hardware type if relevant, e.g.:

OPNsense 24.1.3_1 (amd64).

fichtner commented 2 months ago

At first glance it seems odd that there is an issue with the routes. The obvious: did you check 24.1.4 and the routing table?

StrikerTwo commented 2 months ago

I have the same problem. Worked fine for years before the upgrade. My tunnel network is 10.88.0.0/24. Peers that run Wireguard themselves work fine, but allowed IPs outside the tunnel network (10.0.0.0/24 in this case) do not create routes in OPNsense, and clients on that network cannot reach either the OPNsense node or others behind it.

I mitigated the situation by manually adding a static route, then everything instantly works again. Come to think about it, I already had the peers registered as gateways before so I could monitor the connection on the dashboard - maybe that is the problem? Does it stop adding the route because the gateway is already there?

interj commented 2 months ago

I have upgraded to 24.1.5_3 and rebooted with the extra gateway and route disabled - hoping it might generate them after the boot. Unfortunately behaviour is the same - if I don't have the gateway and static route enabled in the config no route to 192.168.1.0/24 gets added, as soon as I enable them again I can see the route with netstat -rn.

I found something else though - plugin os-wireguard (missing) highlighted in red shows in the firmware/plugins tab, trying to install it yields this result:

***GOT REQUEST TO INSTALL***
Currently running OPNsense 24.1.5_3 at Mon Apr  8 17:30:18 CEST 2024
Updating OPNsense repository catalogue...
OPNsense repository is up to date.
All repositories are up to date.
pkg: No packages available to install matching 'os-wireguard' have been found in the repositories
Checking integrity... done (0 conflicting)
Nothing to do.
***DONE***
fichtner commented 2 months ago

Just check if you have assigned wg interfaces and if so that the IPv4 mode is ‚none‘ and if it is not go and fix this by moving the IP address to the instance tunnel address setting.

interj commented 2 months ago

Ok, setting image on wireguard interface solves the issue, routes are added automatically. But if I add tunnel address as an extra gateway (for monitoring the tunnel for example, as @StrikerTwo mentioned) this prevents the routes from being generated.

StrikerTwo commented 2 months ago

Can confirm it works if I remove both the gateways and the IP configuration on the interface.

Was there a manual that I missed? Because it used to work like this with the old plugin. The current documentation is slightly misleading as well:

The tunnel addresses are configured on the wgX device (which is always visible in Interfaces ‣ Overview).

For example when choosing 10.10.0.1/24 the wgX instance has this address configured

It doesn't explicitly say "and under no circumstances do this configuration on the interface directly" ;)

fichtner commented 2 months ago

Validation was added on the interface settings page on our way to 24.1 to avoid configuring IPvX modes for WireGuard tunnels in alignment with all other tunnels. This may have been a source of problems in the past as well when wireguard was just a plugin contributed by @mimugmail in a community scope and had to be reworked for reliable core integration.

justyns commented 1 month ago

So FWIW, I am working on setting up a wireguard site-to-site vpn between two opnsense machines.

On one machine, version OPNsense 24.1.7, it did not automatically create routes for the wireguard Allowed IPs cidrs.

On the other machine, version OPNsense 24.1.6, it did automatically create routes for all of the Allowed IPs.

As far as I can tell, the config related to wireguard should be almost the same between the two sites. I didn't see any wireguard packages changing between 24.1.6 and 24.1.7 changing. But both versions are newer than OP's.


edit: Actually as I finished typing this, I just tried disabling and enabling the peer on the 24.1.7 machine. That was apparently enough to fix it and cause it to create the missing routes. I suspect updating Allowed IPs after the peer is created doesn't cause the routes to get updated?