This is not an issue more of a "is this possible" question about routing to the VPN

[sudo-kraken]

I was hoping to have the UDM Pro connect to a open vpn connection, then have certain vlans in my network have access to this without the use of a vpn client but still be able to function as normal, ie still be able to talk to other local vlans and devices as well as be able to route to endpoints on the other end of the vpn, is this possible?

If it is not possible what is the best way for me to achieve something close to this? I just want to be able to access the vpn from multiple vlans without disrupting current network config but allow them out to this new vpn connection without the need for a client.

[peacey]

Hi @Joeharrison94,

Yes, your use case is exactly what this script is for.

Just:

1. Follow the instructions to setup the script with OpenVPN
2. Configure your vpn.conf to force particular VLAN interfaces with FORCED_SOURCE_INTERFACE or VLAN subnets with FORCED_SOURCE_IPV4. This will force those VLAN's traffic through the VPN tunnel.
3. Add exemptions to your local vlan subnets by adding them to EXEMPT_DESTINATIONS_IPV4 in your vpn.conf. This will exempt those local subnets from going out the VPN tunnel.

[sudo-kraken]

Hi @Joeharrison94,

Yes, your use case is exactly what this script is for.

Just:

1. Follow the instructions to setup the script with OpenVPN
2. Configure your vpn.conf to force particular VLAN interfaces with FORCED_SOURCE_INTERFACE or VLAN subnets with FORCED_SOURCE_IPV4. This will force those VLAN's traffic through the VPN tunnel.
3. Add exemptions to your local vlan subnets by adding them to EXEMPT_DESTINATIONS_IPV4 in your vpn.conf. This will exempt those local subnets from going out the VPN tunnel.

So is this for forcing the entire vlan down the tunnel and then exempting some IPS to not go down the tunnel.

Or is it possible to say allow my mgmt machine to access everything as normal and also be able to connect to the VPN endpoints also to have access to both?

[peacey]

It is possible to do it the other way, but then you don't have control over overriding UDMP subnets to go through the tunnel. Also I haven't really tested it that way extensively.

If you do want to try it this way, then in your vpn.conf, set PREF to a number between 32001 and 32499 (for example PREF=32100). This will make the VPN routing rule apply AFTER the local subnet routing rules (which has pref of 32000) but before WAN routing rules (which has pref 35000). Hence your device will be able to communicate with all local subnets and anything that's not local will go out the VPN.

You will still need to set the FORCEDSOURCE* variables for VLAN forcing. But you don't need to use the EXEMPT anymore for local subnets. But remember in this case, trying to force a local UDMP subnet as a FORCED_DESTINATION will not work anymore. I know you don't care about that, but just an important note to remember.

Try it out and tell me if it works for you. As I said, I haven't tested it extensively this way, so tell me if you encounter any weird issues.

[sudo-kraken]

Cool thank you for this, I may have some further setup questions when I look at this tomorrow but hopefully I can work it out and test I, will I need to put any routing rules in place to be able to reach VPN endpoints i.e. my mgmt machine on vlan 40 on 10.0.40.45 will be able to reach everything as normal but when it wants to hit 192.168.100.50 over the VPN will it know how to get there.

[peacey]

As long as 192.168.100.0/24 is not a UDMP local subnet, the traffic will go out the VPN tunnel if you added the vlan to FORCEDSOURCE*. You won't have to add any routing rules yourself. By default, everything goes through the tunnel.

One thing to note is by default the script adds masquerading on the tunnel so that any outgoing packets through the tunnel have their source set to the UDMP VPN IP instead of UDMP client IP (this is necessary for 3rd party VPN providers). However if you plan to setup your own OpenVPN server and want to retain the UDMP client IPs in VPN traffic, you can disable masquerading in the script by setting BYPASS_MASQUERADE_IPV4="ALL" in your vpn.conf. If you disable masquerading, you will have to ensure your OpenVPN server has the correct routes in place for the UDMP subnets to go back through the VPN tunnel on the server-side (by configuring iroute option in OpenVPN for example).

[sudo-kraken]

It is possible to do it the other way, but then you don't have control over overriding UDMP subnets to go through the tunnel. Also I haven't really tested it that way extensively.

If you do want to try it this way, then in your vpn.conf, set PREF to a number between 32001 and 32499 (for example PREF=32100). This will make the VPN routing rule apply AFTER the local subnet routing rules (which has pref of 32000) but before WAN routing rules (which has pref 35000). Hence your device will be able to communicate with all local subnets and anything that's not local will go out the VPN.

You will still need to set the FORCEDSOURCE* variables for VLAN forcing. But you don't need to use the EXEMPT anymore for local subnets. But remember in this case, trying to force a local UDMP subnet as a FORCED_DESTINATION will not work anymore. I know you don't care about that, but just an important note to remember.

Try it out and tell me if it works for you. As I said, I haven't tested it extensively this way, so tell me if you encounter any weird issues.

I also have a question about putting rules after my defined ones my final rule after my allow rules is a deny all do I need to put the preference number to before that rule?

[sudo-kraken]

That's in Lan in

[peacey]

No @Joeharrison94, those are firewall rules not routing rules and are not related. I am talking about routing rules which you can see by executing ip rule in SSH. These routing rules define lookup by table number or name.

There's a main rule at pref 32000, which is the main table that has local subnets. Then there's the WAN rules (lookup 201/202) at 32500+.

[sudo-kraken]

No @Joeharrison94, those are firewall rules not routing rules and are not related. I am talking about routing rules which you can see by executing ip rule in SSH. These routing rules define lookup by table number or name.

There's a main rule at pref 32000, which is the main table that has local subnets. Then there's the WAN rules (lookup 201/202) at 32500+.

Understood, so the 192.168.100.0/24 isn't a local subnet, basically my vlans are vlan 40 and 80 each with their own subnets, I can put the rules in after mine and everything in those vlans can talk as normal across local vlans and retain Thier internet access locally but also route down the VPN also. Thanks for the help btw this is the most amazing project btw.

[peacey]

Yes that's right. Just set PREF=32100 and set FORCED_SOURCE_INTERFACE="br40 br80" to force vlan 40 and 80 and you should be good to go. And take note of the masquerading as I explained above in case you want to disable that.

[sudo-kraken]

Yes that's right. Just set PREF=32100 and set FORCED_SOURCE_INTERFACE="br40 br80" to force vlan 40 and 80 and you should be good to go. And take note of the masquerading as I explained above in case you want to disable that.

I have successfully managed to get this working using the guides I only changed the above settings you listed in the vpn conf, however I am getting this error "2022-03-07 19:46:34 tun packet too large on write (tried=1500,max=1448)", and my internet speed on those vlans drops from 900mbps to 9-12mbps which is a bit odd, the second it is turned off back to full throughput. I also tried to run it using the scripts but it doesnt seem to work it only work when you run it with the command line args? any thoughts, I will look at it in greater detail tomorrow but wondered if you have any ideas.

[peacey]

This sounds like an MTU/MSS issue with your VPN server. Did you set up the VPN server yourself? I would set the MSS clamping to 1400 on the server and client. On both openvpn server and client configs, try to add the option mssfix 1400.

Alternatively, you can also try to set MSS_CLAMPING_IPV4="1400" in your vpn.conf config. But I'm not sure if changing it only on client and not server will work, so try both ways.

Also, keep in mind OpenVPN routing is limited to ~200 Mbps due to CPU limitations. If you need anything faster than that, you'll need to use wireguard which can reach ~700 Mbps.

Script should work fine. Double check you did it right. I would try to get it working normally without the script first and we can deal with the script afterwards.

[sudo-kraken]

This sounds like an MTU/MSS issue with your VPN server. Did you set up the VPN server yourself? I would set the MSS clamping to 1400 on the server and client. On both openvpn server and client configs, try to add the option mssfix 1400.

Alternatively, you can also try to set MSS_CLAMPING_IPV4="1400" in your vpn.conf config. But I'm not sure if changing it only on client and not server will work, so try both ways.

Also, keep in mind OpenVPN routing is limited to ~200 Mbps due to CPU limitations. If you need anything faster than that, you'll need to use wireguard which can reach ~700 Mbps.

Script should work fine. Double check you did it right. I would try to get it working normally without the script first and we can deal with the script afterwards.

No it's not my server but I can get this set though on both ends and no problem re speeds I'm happy with the 200 but not sure why enabling it drops my wan connection down to 9mbps usually with the openvpn client on the machine locally it's still a split tunnel and full wan speed

[peacey]

If you don't have access to the server, you can first try to set MSS_CLAMPING_IPV4="1400" in vpn.conf and mssfix 1400 in openvpn client config. If that doesn't help, then try to change the server. You might not need to change the server.

MTU issues can definitely cause severe slowdowns due to fragmentation and retries. So better try the MSS fix.

Not sure why it works on your local machine without changing MSS, honestly. MSS issues are a mystery.

[sudo-kraken]

If you don't have access to the server, you can first try to set MSS_CLAMPING_IPV4="1400" in vpn.conf and mssfix 1400 in openvpn client config. If that doesn't help, then try to change the server. You might not need to change the server.

MTU issues can definitely cause severe slowdowns due to fragmentation and retries. So better try the MSS fix.

Not sure why it works on your local machine without changing MSS, honestly. MSS issues are a mystery.

MTU issue is sorted, the thing that I need to sort is that everything on those two vlans internet connection drops to sub 10 mbps from 920mbps when the vpn is enabled for internet bound traffic which should not go down the vpn. (note that the speedtest on the udm pro itself is unaffected that still hits normal speeds only the two vlans are slowed)

[sudo-kraken]

@peacey all mtu issues resolved, but the split tunnel seems to be majorly hampering wan internet connections from 920mbps to 9mbps which is not routed down the tunnel

[peacey]

@Joeharrison94, the script shouldn't force anything down the VPN other than the VLANs you added to FORCEDSOURCE* options. You're saying you didn't add a VLAN to be forced, but the speed of that VLAN is still being impacted? That shouldn't happen since the script doesn't touch anything outside of the VLANs you forced.

[sudo-kraken]

@peacey No so I force a vlan down the tunnel but with pref 32100 so it should not effect the wan traffic or its speed but it is doing, it is still accessible but its speed is next to nothing

[sudo-kraken]

vpn.ovpn

remote omitted 1194
auth-user-pass username_password.txt
float
nobind
proto udp
dev tun
sndbuf 0
rcvbuf 0
keepalive 15 60
comp-lzo adaptive
client
auth SHA256
cipher AES-256-CBC
#link-mtu 1570
tun-mtu 1500
mssfix 1500
ns-cert-type server
<ca>
-----BEGIN CERTIFICATE-----
omitted
-----END CERTIFICATE-----

</cert>

<key>
-----BEGIN PRIVATE KEY-----
omitted
-----END PRIVATE KEY-----

vpn.conf

### SPLIT VPN OPTIONS ###
# Enter multiple entries separated by spaces.
# Do not enter square brackets around the entries.

# Force these sources through the VPN.
# Format: [brX] for interface. [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
FORCED_SOURCE_INTERFACE=""
FORCED_SOURCE_IPV4="10.0.180.5"
FORCED_SOURCE_IPV6=""
FORCED_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
FORCED_SOURCE_IPV4_PORT=""
FORCED_SOURCE_IPV6_PORT=""
FORCED_SOURCE_MAC_PORT=""

# Force these destinations through the VPN.
# These destinations will be forced regardless of source.
# Format: [IP/nn]
FORCED_DESTINATIONS_IPV4=""
FORCED_DESTINATIONS_IPV6=""

# Force local UDM traffic going out of these WAN interfaces to go through the
# VPN instead for both IPv4 and IPv6 traffic.
# This does not include routed traffic, only local traffic generated by the UDM.
# Do not enable this unless you want to force UDM local traffic through the VPN.
# For UDM-Pro, set to "eth8" for WAN1/Ethernet port, or "eth9" for WAN2/SFP+ port,
# or "eth8 eth9" for both. For UDM Base, set to "eth1" for the WAN port.
# This option might cause unintended problems, so disable it if you encounter any issues.
FORCED_LOCAL_INTERFACE=""

# Exempt these sources from the VPN.
# Format: [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
EXEMPT_SOURCE_IPV4=""
EXEMPT_SOURCE_IPV6=""
EXEMPT_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
EXEMPT_SOURCE_IPV4_PORT=""
EXEMPT_SOURCE_IPV6_PORT=""
EXEMPT_SOURCE_MAC_PORT=""

# Exempt these destinations from the VPN.
# Format: [IP/nn]
EXEMPT_DESTINATIONS_IPV4=""
EXEMPT_DESTINATIONS_IPV6=""

# Force/exempt these IP sets
# IP sets need to be created before this script is run or the script will error.
# IP sets can be updated externally and will be matched dynamically.
# Each IP set entry consists of the IP set name and whether to match on source
# or destination. src/dst needs to be specified for each IP set field.
#
# Enable NAT hairpin by exempting UBIOS_ADDRv4_ethX:dst for IPv4 or
# UBIOS_ADDRv6_ethX:dst for IPv6 (where X = 8 for RJ45, or 9 for SFP+ WAN).
# For IPv6 prefix delegation, exempt UBIOS_ADDRv6_brX, where X = VLAN number (0 = LAN).
#
# To allow communication with your VLAN subnets without hardcoding the subnets,
# exempt the UBIOS_NETv4_brX:dst ipset for IPv4 or UBIOS_NETv6_brX:dst for IPv6.
#
# Format: [IPSet Name]:[src/dst,src/dst,...]
FORCED_IPSETS=""
EXEMPT_IPSETS=""

# VPN port forwards.
# Format: [tcp/udp/both]-[VPN Port]-[Forward IP]-[Forward Port]
PORT_FORWARDS_IPV4=""
PORT_FORWARDS_IPV6=""

# Redirect IPv4 and IPv6 DNS to these addresses for VPN-destined traffic.
# Note that many VPN providers redirect DNS going through their VPN network
# to their own DNS servers. Redirection to other IPs might not work on all providers,
# except for DNS redirects to a local address, or rejecting DNS traffic completely.
#
# IPV4 Format: [IP] to redirect to IP, "DHCP" if using OpenVPN or OpenConnect to obtain
# DNS from DHCP options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types like wireguard/external.
#
# Example: Get DNS from DHCP
DNS_IPV4_IP="DHCP"
DNS_IPV4_PORT=53
# Set this to the interface (brX) the DNS is on if it is a local IP. Leave blank for
# non-local IPs. Local DNS redirects will not work without specifying the interface.
DNS_IPV4_INTERFACE=""

# IPV6 Format: [IP] to redirect to IP, or "REJECT" to reject IPv6 DNS traffic completely.
# IPV6 Format: [IP] to redirect to IP, "DHCP" if using OpenConnect to obtain DNS from DHCP
# options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types.
DNS_IPV6_IP=""
DNS_IPV6_PORT=53
DNS_IPV6_INTERFACE=""

# Bypass masquerade (SNAT) for these source IPs. This option should only be used if your
# VPN server is setup to know how to route the subnet you do not want to masquerade
# (e.g.: the "iroute" option in OpenVPN).
# Set these options to ALL to disable masquerading completely.
# Format: [IP/nn] or "ALL"
BYPASS_MASQUERADE_IPV4=""
BYPASS_MASQUERADE_IPV6=""

# Enabling kill switch drops VPN-destined traffic that doesn't go through the VPN.
KILLSWITCH=0

# Enable this only if you are testing or you don't care about your real IP leaking
# when the vpn client restarts or exits.
REMOVE_KILLSWITCH_ON_EXIT=1

# Enable this if you added blackhole routes in the Unifi Settings to prevent Internet
# access at system startup before the VPN script runs. This option removes the blackhole
# routes to restore Internet access after the killswitch has been enabled.
# If you do not set this to 1, openvpn will not be able to connect at startup, and your
# Internet access will never be enabled until you manually remove the blackhole routes.
# Set this to 0 only if you did not add any blackhole routes.
REMOVE_STARTUP_BLACKHOLES=1

# Set the VPN provider.
# "openvpn" for OpenVPN (default), "openconnect" for OpenConnect, "external" for wireguard,
# or "nexthop" for an external VPN client.
VPN_PROVIDER="openvpn"

# If using "external" for VPN_PROVIDER, set this to the VPN endpoint IP so that the
# gateway route can be automatically added for the VPN endpoint.
# OpenVPN passes the VPN endpoint IP to the script and will override these values.
# These must be defined if using VPN_PROVIDER="nexthop".
VPN_ENDPOINT_IPV4=""
VPN_ENDPOINT_IPV6=""

# Set this to the route table that contains the gateway route, "auto", or "disabled".
# The Ubiquiti route table is "201" if you're using Ethernet, "202" for SFP+, and
# "203" for U-LTE.
# Default is "auto" which works with WAN failover and automatically changes the endpoint
# via gateway route when the WAN or gateway routes changes.
# Set to "disabled" if you are using the nexthop option to connect to a VPN on your LAN.
GATEWAY_TABLE="auto"

# Set the MSS clamping on packets going out the VPN tunnel. Usually, it is not needed to
# set this manually, but some VPN connections stall if the MSS clamping is not set correctly.
# Typical values range from 1240 to 1460, but it could be lower.
MSS_CLAMPING_IPV4="1500"
MSS_CLAMPING_IPV6=""

# Set this to the timer to use for the rule watcher (in seconds).
# The script will wake up every N seconds to re-add rules if they're deleted by
# the system, or change gateway routes if they changed. Default is 1 second.
WATCHER_TIMER=1

# Options for custom table and chains.
# These options need to be unique for each instance of openvpn if running multiple.
ROUTE_TABLE=101
MARK=0x9
PREFIX="VPN_"
PREF=32100
DEV=tun0

# To execute commands when the VPN connects or disconnects, you can use the
# callback functions hooks_pre_up, hooks_up, hooks_down, and
# hooks_force_down. These functions will be invoked in response to VPN events
# pre-up, up, down, and force-down respectively.
#
# For an example on using these hooks, please see vpn.conf.filled.sample.
#

run-vpn.sh

#!/bin/sh
# Load configuration and run openvpn
cd /etc/split-vpn/openvpn/vpn
. ./vpn.conf
# /etc/split-vpn/vpn/updown.sh ${DEV} pre-up >pre-up.log 2>&1
nohup openvpn --config vpn.ovpn \
              --route-noexec --redirect-gateway def1 \
              --up /etc/split-vpn/vpn/updown.sh \
              --down /etc/split-vpn/vpn/updown.sh \
              --dev-type tun --dev ${DEV} \
              --script-security 2 \
              --mute-replay-warnings >openvpn.log 2>&1 &

[peacey]

So in your vpn.conf, you only forced client 10.0.180.5. So any WAN (Internet) traffic initiated from 10.0.180.5 will be forced down the VPN, and the speed will be affected due to going through the VPN. Right?

Are you saying other clients that are not forced are going through the VPN and having their Internet speed affected too?

[sudo-kraken]

No so I just changed it to effect one machine in the vlan 10.0.180.5, and everything else in that vlan is fine speed is full and working, the 10.0.180.5 is very very slow on the wan, so speedtest.net reports a massive drop, but speeds on the vpn work fine as they are throttled to 2mbps anyway as its for an sftp. The issue I mean is that if I run an openvpn client on the machine with the same settings my internet speed remains as normal at 920mbps and the tunnel traffic is fine as normal at 2mbps.

I thought it would function the same as a split vpn so wan is normal and vpn is vpn bound as it cant be using the vpn for internet access as the vpn is just an sftp without internet access on the other end of the vpn so it cant be using the vpn connection to access the internet.

[peacey]

Ah. Sorry @Joeharrison94, I misunderstood what you wanted to do from the beginning. I thought you wanted to push the WHOLE internet of the forced clients through the VPN, not just VPN endpoints.

By default the script forces all internet through the VPN (because it was made for use with 3rd party VPN providers). If you want to force only VPN endpoints, you need to disable the default routes and blackholes routes by adding these options to your vpn.conf (these options don't exist by default, you have to add them yourself anywhere in the file):

DISABLE_DEFAULT_ROUTE=1
DISABLE_BLACKHOLE=1

Also, MSS_CLAMPING_IPV4="1500" is not valid. MSS should be 1460 or less (MSS is MTU-40). It cannot be 1500. I thought we set it to 1400 to fix the MTU issue, anyways?

Finally, double-check that OpenVPN added the correct VPN routes by running ip route show table 101 after you start the VPN.

FYI, for what you want to do, the only benefit to split-vpn is that you can control which clients are able to communicate over the VPN. If you wanted all clients on your network to be able to communicate with the VPN and didn't want to selectively choose, you don't even need this script and can just run openvpn alone.

[sudo-kraken]

@peacey What I wanted to achieve was a vlan wide vpn connection that didnt effect internet access or local rules e.g. all machines can operate as normal but also access the remote vpn sftp without the need for a client that 10.0.180.5 is a test box as it hampered my whole vlan internet speed. So is it possible to have the aforementioned so everything can reach the sftp on the vpn without a client and also have full internet speed and access to the local network.

[peacey]

Yes @Joeharrison94, as I said it's possible if you add the two options I mentioned above (DISABLE_DEFAULT_ROUTE and DISABLE_BLACKHOLE).

Add those options and then run the VPN and show me the output of ip route show table 101 so we can confirm OpenVPN is adding the correct VPN routes.

[sudo-kraken]

@peacey

# ip route show table 101
10.8.0.1 via 10.8.0.5 dev tun0
82.43.119.61 via 82.36.40.1 dev eth8
192.168.100.0/24 via 10.8.0.5 dev tun0

So now I am getting decent internet download speeds, and it can ping the sftp on 192.168.100.21 but it cant connect to it on WinSCP

[sudo-kraken]

@peacey none of these are my public ip though I believe it is that of the vpn

[sudo-kraken]

Just started working now for some reason I can now connect to the vpn :)

[peacey]

Did you set MSS_CLAMPING_IPV4="1400" not 1500? Also, in the client openvpn you should do mssfix 1400 not 1500.

That's great! So everything is good then? Try to set it for the whole VLAN now.

[sudo-kraken]

@peacey I will set it for the whole vlan now, and I will set those values also, in the ovpn file should I set tun-mtu to 1400 or 1500 if the other two values are 1400

[sudo-kraken]

@peacey Set those values as mentioned above and tun-mtu to 1500 and its working like a dream only drops wan to about 800mbps which Im more than happy to live with! any other recommendations for settings? and thank you

[peacey]

Your WAN speed isn't going to be affected because it's not going through the VPN (other than for the VPN subnets). The MTU/MSS is for speed on the VPN. Actually, you probably shouldn't set tun-mtu, only set mssfix.

Those settings are enough for your use case. No need for anything else unless you want to do something else.

You're welcome!

[sudo-kraken]

@peacey I willi remove tun-mtu then and thanks so much, what are the other main use cases for this just incase I want to branch out, and if I wanted to use some other use cases would me running adguard on my router be an issue in its own vlan?

[peacey]

The main use case of this script is to route Internet traffic for some clients through a VPN provider like NordVPN or Mullvad. For example, forcing an Apple TV to be on the VPN so you watch videos from streaming services in other countries.

If the AdGuard is in its own vlan that isn't forced, then it won't be affected by the script. It won't cause any issues.

[sudo-kraken]

@peacey awesome that would be good maybe create another one in a new vlan for media devices and have them route to other countries, will it matter that everything in my network is told to use the adguard for dns, I was also thinking of adding scripts to force anything with hardcoded dns to use adguard also.

[peacey]

No it won't matter. It should all work fine together.

[sudo-kraken]

legend thank you again I will close this issue now.