forced source based on interface (br2)

[david00769]

For the life of me I can't seem to get this to work and am stuck - would appreciate any help.

Scenario - Site to Site VPN established using Unifi's open VPN feature and is tested as working. I logged on to router, and tested ping of all available interfaces (side one of tunnel is 10.8.8.1, other side is 10.8.8.2). Can also ping remote gateway and access web pages on devices at both ends of the tunnel from the router.

When I log onto the wifi network, it works (until I bring the tunnel upland then traffic stops).

I have a separate wifi network (192.168.3.0/24) on interface br2 that I would like to fource everything through from that network the default gateway on the other side of the tunnel.

Here are my logs...

# ip route show
10.0.0.0/24 via 10.8.8.2 dev tun1 
10.0.1.0/24 via 10.8.8.2 dev tun1 
10.8.8.2 dev tun1 proto kernel scope link src 10.8.8.1 
167.XXX.XXX.XXX/22 dev eth8 proto kernel scope link src 167.XXX.XXX.XXX
192.168.3.0/24 dev br2 proto kernel scope link src 192.168.3.1 
192.168.100.0/24 dev br0 proto kernel scope link src 192.168.100.1 

# iptables -t mangle -S | grep VPN
-N VPN_FORWARD
-N VPN_OUTPUT
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A FORWARD -j VPN_FORWARD
-A OUTPUT -j VPN_OUTPUT
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_PREROUTING -i br2 -j MARK --set-xmark 0x169/0xffffffff
-A VPN_PREROUTING -d 10.8.8.1/32 -m mark --mark 0x169 -j MARK --set-xmark 0x0/0xffffffff
# iptables -t nat -S | grep VPN
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A POSTROUTING -j VPN_POSTROUTING
# iptables -t filter -S | grep VPN
-N VPN_FORWARD
-N VPN_INPUT
-A INPUT -j VPN_INPUT
-A FORWARD -j VPN_FORWARD
# ip6tables -t mangle -S | grep VPN
-N VPN_FORWARD
-N VPN_OUTPUT
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A FORWARD -j VPN_FORWARD
-A OUTPUT -j VPN_OUTPUT
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_PREROUTING -i br2 -j MARK --set-xmark 0x169/0xffffffff
# ip6tables -t nat -S | grep VPN
-N VPN_POSTROUTING
-N VPN_PREROUTING
-A PREROUTING -j VPN_PREROUTING
-A POSTROUTING -j VPN_POSTROUTING
-A VPN_POSTROUTING -o tun1 -m mark --mark 0x169 -j MASQUERADE
# ip6tables -t filter -S | grep VPN
-N VPN_FORWARD
-N VPN_INPUT
-A INPUT -j VPN_INPUT
-A FORWARD -j VPN_FORWARD

# ip route show table 101
0.0.0.0/1 via 10.8.8.2 dev tun1 
blackhole default 
128.0.0.0/1 via 10.8.8.2 dev tun1

# ip -6 route show table 101
blackhole default dev lo metric 1024 pref medium

# ip rule
0:  from all lookup local 
99: from all fwmark 0x169 lookup 101 
32000:  from all lookup main 
32500:  from 167.179.139.97 lookup 201 
32766:  from all lookup 201 
32767:  from all lookup default 

# ip -6 rule
0:  from all lookup local 
99: from all fwmark 0x169 lookup 101 
32000:  from all lookup main 
32501:  from fe80::72a7:41ff:feed:ccc9 lookup 201 
32766:  from all lookup 201 

# ps | grep updown.sh
12268 root     grep updown.sh
22248 root     {updown.sh} /bin/sh /etc/split-vpn/vpn/updown.sh tun1 up site1
22251 root     {updown.sh} /bin/sh /etc/split-vpn/vpn/updown.sh tun1 up site1

Here is the config file I am using

# cat vpn.conf

### SPLIT VPN OPTIONS ###
# Enter multiple entries separated by spaces.
# Do not enter square brackets around the entries.

# Force these sources through the VPN.
# Format: [brX] for interface. [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
FORCED_SOURCE_INTERFACE="br2"
FORCED_SOURCE_IPV4=""
FORCED_SOURCE_IPV6=""
FORCED_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
FORCED_SOURCE_IPV4_PORT=""
FORCED_SOURCE_IPV6_PORT=""
FORCED_SOURCE_MAC_PORT=""

# Force these destinations through the VPN. 
# These destinations will be forced regardless of source.
# Format: [IP/nn]
FORCED_DESTINATIONS_IPV4=""
FORCED_DESTINATIONS_IPV6=""

# Force local UDM traffic going out of these WAN interfaces to go through the
# VPN instead for both IPv4 and IPv6 traffic.
# This does not include routed traffic, only local traffic generated by the UDM.
# Do not enable this unless you want to force UDM local traffic through the VPN.
# For UDM-Pro, set to "eth8" for WAN1/Ethernet port, or "eth9" for WAN2/SFP+ port, 
# or "eth8 eth9" for both. For UDM Base, set to "eth1" for the WAN port.
# This option might cause unintended problems, so disable it if you encounter any issues.
FORCED_LOCAL_INTERFACE=""

# Exempt these sources from the VPN. 
# Format: [IP/nn] for IP. [xx:xx:xx:xx:xx:xx] for mac.
EXEMPT_SOURCE_IPV4=""
EXEMPT_SOURCE_IPV6=""
EXEMPT_SOURCE_MAC=""

# Format: [tcp/udp/both]-[IP/MAC Source]-[port1,port2:port3,port4,...]
# Maximum 15 ports per entry.
EXEMPT_SOURCE_IPV4_PORT=""
EXEMPT_SOURCE_IPV6_PORT=""
EXEMPT_SOURCE_MAC_PORT=""

# Exempt these destinations from the VPN. 
# Format: [IP/nn]
EXEMPT_DESTINATIONS_IPV4=""
EXEMPT_DESTINATIONS_IPV6=""

# Force/exempt these IP sets
# IP sets need to be created before this script is run or the script will error.
# IP sets can be updated externally and will be matched dynamically.
# Each IP set entry consists of the IP set name and whether to match on source
# or destination. src/dst needs to be specified for each IP set field.
#
# Enable NAT hairpin by exempting UBIOS_ADDRv4_ethX:dst for IPv4 or
# UBIOS_ADDRv6_ethX:dst for IPv6 (where X = 8 for RJ45, or 9 for SFP+ WAN).
# For IPv6 prefix delegation, exempt UBIOS_ADDRv6_brX, where X = VLAN number (0 = LAN).
#
# To allow communication with your VLAN subnets without hardcoding the subnets,
# exempt the UBIOS_NETv4_brX:dst ipset for IPv4 or UBIOS_NETv6_brX:dst for IPv6.
#
# Format: [IPSet Name]:[src/dst,src/dst,...]
FORCED_IPSETS=""
EXEMPT_IPSETS=""

# VPN port forwards.
# Format: [tcp/udp/both]-[VPN Port]-[Forward IP]-[Forward Port]
PORT_FORWARDS_IPV4=""
PORT_FORWARDS_IPV6=""

# Redirect IPv4 and IPv6 DNS to these addresses for VPN-destined traffic.
# Note that many VPN providers redirect DNS going through their VPN network
# to their own DNS servers. Redirection to other IPs might not work on all providers,
# except for DNS redirects to a local address, or rejecting DNS traffic completely.
#
# IPV4 Format: [IP] to redirect to IP, "DHCP" if using OpenVPN or OpenConnect to obtain
# DNS from DHCP options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types like wireguard/external.
#
# Example: Get DNS from DHCP
DNS_IPV4_IP="DHCP"
DNS_IPV4_PORT=53
# Set this to the interface (brX) the DNS is on if it is a local IP. Leave blank for
# non-local IPs. Local DNS redirects will not work without specifying the interface.
DNS_IPV4_INTERFACE=""

# IPV6 Format: [IP] to redirect to IP, or "REJECT" to reject IPv6 DNS traffic completely.
# IPV6 Format: [IP] to redirect to IP, "DHCP" if using OpenConnect to obtain DNS from DHCP
# options, or "REJECT" to reject all DNS traffic. "DHCP" is not supported on
# other VPN types.
DNS_IPV6_IP=""
DNS_IPV6_PORT=53
DNS_IPV6_INTERFACE=""

# Bypass masquerade (SNAT) for these source IPs. This option should only be used if your 
# VPN server is setup to know how to route the subnet you do not want to masquerade 
# (e.g.: the "iroute" option in OpenVPN).
# Set these options to ALL to disable masquerading completely.
# Format: [IP/nn] or "ALL"
BYPASS_MASQUERADE_IPV4="ALL"
BYPASS_MASQUERADE_IPV6=""

# Enabling kill switch drops VPN-destined traffic that doesn't go through the VPN.
KILLSWITCH=0

# Enable this only if you are testing or you don't care about your real IP leaking
# when the vpn client restarts or exits.
REMOVE_KILLSWITCH_ON_EXIT=1

# Enable this if you added blackhole routes in the Unifi Settings to prevent Internet
# access at system startup before the VPN script runs. This option removes the blackhole 
# routes to restore Internet access after the killswitch has been enabled. 
# If you do not set this to 1, openvpn will not be able to connect at startup, and your 
# Internet access will never be enabled until you manually remove the blackhole routes. 
# Set this to 0 only if you did not add any blackhole routes.
REMOVE_STARTUP_BLACKHOLES=1

# Set the VPN provider.
# "openvpn" for OpenVPN (default), "openconnect" for OpenConnect, "external" for wireguard,
# or "nexthop" for an external VPN client.
VPN_PROVIDER="nexthop"

# If using "external" for VPN_PROVIDER, set this to the VPN endpoint IP so that the
# gateway route can be automatically added for the VPN endpoint.
# OpenVPN passes the VPN endpoint IP to the script and will override these values.
# These must be defined if using VPN_PROVIDER="nexthop".
VPN_ENDPOINT_IPV4="10.8.8.2"
VPN_ENDPOINT_IPV6=""

# Set this to the route table that contains the gateway route, "auto", or "disabled".
# The Ubiquiti route table is "201" if you're using Ethernet, "202" for SFP+, and
# "203" for U-LTE.
# Default is "auto" which works with WAN failover and automatically changes the endpoint
# via gateway route when the WAN or gateway routes changes.
# Set to "disabled" if you are using the nexthop option to connect to a VPN on your LAN.
GATEWAY_TABLE="disabled"

# Set the MSS clamping on packets going out the VPN tunnel. Usually, it is not needed to
# set this manually, but some VPN connections stall if the MSS clamping is not set correctly.
# Typical values range from 1240 to 1460, but it could be lower.
MSS_CLAMPING_IPV4=""
MSS_CLAMPING_IPV6=""

# Set this to the timer to use for the rule watcher (in seconds).
# The script will wake up every N seconds to re-add rules if they're deleted by
# the system, or change gateway routes if they changed. Default is 1 second.
WATCHER_TIMER=1

# Options for custom table and chains.
# These options need to be unique for each instance of openvpn if running multiple.
ROUTE_TABLE=101
MARK=0x169
PREFIX="VPN_"
PREF=99
DEV=tun1

# To execute commands when the VPN connects or disconnects, you can use the 
# callback functions hooks_pre_up, hooks_up, hooks_down, and 
# hooks_force_down. These functions will be invoked in response to VPN events 
# pre-up, up, down, and force-down respectively.
#
# For an example on using these hooks, please see vpn.conf.filled.sample.
# 

[peacey]

Hi @david00769,

I believe your config looks correct to force devices from br2 (your wifi).

1. What router are you using on the remote end, and what router on the local end?
2. Did you add 192.168.3.0/24 on the remote gateway's site-to-site settings as a remote subnet?
3. Did you make sure your WiFi devices are on br2 and getting IPs in the subnet 192.168.3.0/24?

4. After you bring the split-vpn configuration up, can you test the following pings from a device on br2 (your WiFi device or LAN)?

   ping 1.1.1.1
   ping 10.8.8.2
   ping 10.0.0.1 (or some IP on 10.0.0.0/24)

5. In your vpn.conf, you should also set MSS_CLAMPING_IPV4="1382" (might have to go even lower) or sites might stall when using a site-to-site tunnel. But the ping in (4) should succeed regardless of this setting.

[david00769]

Thanks for your feedback.

1. Router on the local end is a UDM Pro, and the router on the remote end is a UDM

2. I have added 192.168.3.0/24 on the remote gateways’s site to site settings. It appears that did the trick.

I had the main wifi subnet 192.168.100.0/24 configured on the remote site, but forgot to include the 192.168.3.0/24 for the separate wifi network I wanted to route through the VPN.

Awesome getting a 2nd set of eyes on it from you - much appreciated.

Now the only issue I have left to debug, is why it doesn’t run on boot. For some reason, after a reboot the ./99-run-vpn.sh script is not running automatically. If I manually execute it - it brings things up, so script is ok.

I’ll have to troubleshoot the boot service - suspect its not installed correctly.

On Jan 24, 2023, at 10:22 AM, peacey @.***> wrote:

Hi @david00769 https://github.com/david00769,

I believe your config looks correct to force devices from br2 (your wifi).

What router are you using on the remote end, and what router on the local end?

Did you add 192.168.3.0/24 on the remote gateway's site-to-site settings as a remote subnet?

Did you make sure your WiFi devices are on br2 and getting IPs in the subnet 192.168.3.0/24?

After you bring the split-vpn configuration up, can you test the following pings from a device on br2 (your WiFi device or LAN)?

ping 1.1.1.1 ping 10.8.8.2 ping 10.0.0.1 (or some IP on 10.0.0.0/24) — Reply to this email directly, view it on GitHub https://github.com/peacey/split-vpn/issues/156#issuecomment-1401142918, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5NTN2MJSWUO3IAYXZBBHYLWT4HBXANCNFSM6AAAAAAUEKOVDM. You are receiving this because you were mentioned.

[peacey]

Great @david00769. Glad to see that fixed it. You should also set MSS_CLAMPING_IPV4="1382" or you might run into HTTPS sites not loading or stalling (keep this in mind if you don't change the option now).

For the boot script issue, if you already installed the boot service try this to fix it. There is a bug in udm-boot that doesn't enable it automatically. You have to do it manually the first time after installation. Run these commands:

unifi-os shell
systemctl enable udm-boot
systemctl status udm-boot
exit

Once you do that, make sure 99-run-vpn.sh is in /mnt/data/on_boot.d, and make sure it has executable permissions. Then restart and see if it runs at boot.

Before you restart though, delete the openvpn.log found in the same folder as your vpn.conf. Then after you reboot, show me the output of the openvpn.log file so we can see if it did run or not.

If it still doesn't run at boot and openvpn.log wasn't created, show me the output of the following after the reboot.

unifi-os shell
systemctl status udm-boot
exit

[david00769]

Still no luck on the boot service.

Here’s the current output log

unifi-os shell

@.***:/# systemctl status udm-boot ● udm-boot.service - Run On Startup UDM Loaded: loaded (/lib/systemd/system/udm-boot.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Sun 2023-01-29 11:09:13 A EDT; 3min 51s ago Main PID: 72 (code=exited, status=123) CPU: 101ms

Jan 29 11:09:09 ubnt ssh-proxy[72]: curl: (6) Could not resolve: github.com (Cou ld not contact DNS servers) Jan 29 11:09:10 ubnt ssh-proxy[72]: Pouring /mnt/data/.cache/cni-plugins/cni-plu gins-linux-arm64-latest.tgz Jan 29 11:09:13 ubnt ssh-proxy[72]: udm-boot.service: running /mnt/data/on_boot. d/06-cni-bridge.sh Jan 29 11:09:13 ubnt ssh-proxy[72]: udm-boot.service: running /mnt/data/on_boot. d/99-run-vpn.sh Jan 29 11:09:13 ubnt ssh-proxy[72]: [Sun Jan 29 11:09:13 AEDT 2023] split-vpn: t un1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:09:13 ubnt ssh-proxy[72]: Cannot find device "tun1" Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Main process exited, code=exited, status=123/n/a Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Faile d with result 'exit-code'. Jan 29 11:09:13 ubnt systemd[1]: Failed to start Run On Startup UDM. Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Consumed 101ms CPU time.

Have tried to install it manually, but not seeing success unfortunately.

I was going to follow the manual clean up instructions and give it another go.

On Jan 25, 2023, at 3:14 AM, peacey @.***> wrote:

Great @david00769 https://github.com/david00769. Glad to see that fixed it. You should also set MSS_CLAMPING_IPV4="1382" or you might run into HTTPS sites not loading or stalling (keep this in mind if you don't change the option now).

For the boot script issue, if you already installed the boot service try this to fix it. There is a bug in udm-boot that doesn't enable it automatically. You have to do it manually the first time after installation. Run these commands:

unifi-os shell systemctl enable udm-boot systemctl status udm-boot exit Once you do that, make sure 99-run-vpn.sh is in /mnt/data/on_boot.d, and make sure it has executable permissions. Then restart and see if it runs at boot.

If it still doesn't run at boot, show me the output of the following after a reboot.

unifi-os shell systemctl status udm-boot — Reply to this email directly, view it on GitHub https://github.com/peacey/split-vpn/issues/156#issuecomment-1402209721, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5NTN2JR5NE2OPO6G4AVVQLWT75U5ANCNFSM6AAAAAAUEKOVDM. You are receiving this because you were mentioned.

[david00769]

I can get it up and running, but it does not survive a reboot

unifi-os shell

@.***:/# systemctl

UNIT LOAD ACTIVE SUB DESCRIPTION
dev-boot4.device loaded activating tentative /dev/boot4
dev-boot6.device loaded activating tentative /dev/boot6
dev-disk-sda1.device loaded activating tentative /dev/disk/sda1
-.mount loaded active mounted /
data.mount loaded active mounted /data
dev-disk.mount loaded active mounted /dev/disk
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
etc-hostname.mount loaded active mounted /etc/hostname
etc-hosts.mount loaded active mounted /etc/hosts
etc-resolv.conf.mount loaded active mounted /etc/resolv.conf
etc-systemd-system-unifi\x2dcore.service.d-capabilities\x2dworkaround.conf.mount loaded active mounted /etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf etc-unifi\x2dos-ssh_proxy_port.mount loaded active mounted /etc/unifi-os/ssh_proxy_port
etc_host.mount loaded active mounted /etc_host
mnt-persistent.mount loaded active mounted /mnt/persistent
root-.ssh-id_rsa.mount loaded active mounted /root/.ssh/id_rsa
run-.containerenv.mount loaded active mounted /run/.containerenv
srv.mount loaded active mounted /srv
sys-kernel-config.mount loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount loaded active mounted Kernel Debug File System
tmp.mount loaded active mounted /tmp
usr-lib-version.mount loaded active mounted /usr/lib/version
var-log-journal.mount loaded active mounted /var/log/journal
var-opt-unifi-tmp.mount loaded active mounted /var/opt/unifi/tmp
systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
init.scope loaded active running System and Service Manager
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
exim4.service loaded active running LSB: exim Mail Transport Agent
freeswitch.service loaded active running freeswitch
@. loaded active exited PostgreSQL initial setup service
postgresql.service loaded active exited PostgreSQL RDBMS
@. loaded active running PostgreSQL Cluster 9.6-main
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-modules-load.service loaded active exited Load Kernel Modules
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-sysusers.service loaded active exited Create System Users
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
● udm-boot.service loaded failed failed Run On Startup UDM

uid-agent.service loaded active running UID Agent service
ulp-go.service loaded active running ULP-GO
unifi-base-ucore.service loaded active running UniFi Base Controller
unifi-core.service loaded active running UniFi Core
unifi-pion-gw.service loaded active running Unifi Pion Gateway service
unifi.service loaded active running unifi
-.slice loaded active active Root Slice
system-postgresql.slice loaded active active system-postgresql.slice
system-postgresql\x2dcluster.slice loaded active active system-postgresql\x2dcluster.slice
system.slice loaded active active System Slice
user.slice loaded active active User and Session Slice
avahi-daemon.socket loaded active running Avahi mDNS/DNS-SD Stack Activation Socket
dbus.socket loaded active running D-Bus System Message Bus Socket
systemd-initctl.socket loaded active listening initctl Compatibility Named Pipe
systemd-journald-dev-log.socket loaded active running Journal Socket (/dev/log)
systemd-journald.socket loaded active running Journal Socket
basic.target loaded active active Basic System
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network.target loaded active active Network
paths.target loaded active active Paths
slices.target loaded active active Slices
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of Temporary Directories

LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type.

79 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. @.:/# systemctl start udm-boot.service @.:/# systemctl UNIT LOAD ACTIVE SUB DESCRIPTION
dev-boot4.device loaded activating tentative /dev/boot4
dev-boot6.device loaded activating tentative /dev/boot6
dev-disk-sda1.device loaded activating tentative /dev/disk/sda1
-.mount loaded active mounted /
data.mount loaded active mounted /data
dev-disk.mount loaded active mounted /dev/disk
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
etc-hostname.mount loaded active mounted /etc/hostname
etc-hosts.mount loaded active mounted /etc/hosts
etc-resolv.conf.mount loaded active mounted /etc/resolv.conf
etc-systemd-system-unifi\x2dcore.service.d-capabilities\x2dworkaround.conf.mount loaded active mounted /etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf etc-unifi\x2dos-ssh_proxy_port.mount loaded active mounted /etc/unifi-os/ssh_proxy_port
etc_host.mount loaded active mounted /etc_host
mnt-persistent.mount loaded active mounted /mnt/persistent
root-.ssh-id_rsa.mount loaded active mounted /root/.ssh/id_rsa
run-.containerenv.mount loaded active mounted /run/.containerenv
srv.mount loaded active mounted /srv
sys-kernel-config.mount loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount loaded active mounted Kernel Debug File System
tmp.mount loaded active mounted /tmp
usr-lib-version.mount loaded active mounted /usr/lib/version
var-log-journal.mount loaded active mounted /var/log/journal
var-opt-unifi-tmp.mount loaded active mounted /var/opt/unifi/tmp
systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
init.scope loaded active running System and Service Manager
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
exim4.service loaded active running LSB: exim Mail Transport Agent
freeswitch.service loaded active running freeswitch
@. loaded active exited PostgreSQL initial setup service
postgresql.service loaded active exited PostgreSQL RDBMS
@. loaded active running PostgreSQL Cluster 9.6-main
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-modules-load.service loaded active exited Load Kernel Modules
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-sysusers.service loaded active exited Create System Users
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
udm-boot.service loaded active exited Run On Startup UDM
uid-agent.service loaded active running UID Agent service
ulp-go.service loaded active running ULP-GO
unifi-base-ucore.service loaded active running UniFi Base Controller
unifi-core.service loaded active running UniFi Core
unifi-pion-gw.service loaded active running Unifi Pion Gateway service
unifi.service loaded active running unifi
-.slice loaded active active Root Slice
system-postgresql.slice loaded active active system-postgresql.slice
system-postgresql\x2dcluster.slice loaded active active system-postgresql\x2dcluster.slice
system.slice loaded active active System Slice
user.slice loaded active active User and Session Slice
avahi-daemon.socket loaded active running Avahi mDNS/DNS-SD Stack Activation Socket
dbus.socket loaded active running D-Bus System Message Bus Socket
systemd-initctl.socket loaded active listening initctl Compatibility Named Pipe
systemd-journald-dev-log.socket loaded active running Journal Socket (/dev/log)
systemd-journald.socket loaded active running Journal Socket
basic.target loaded active active Basic System
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network.target loaded active active Network
paths.target loaded active active Paths
slices.target loaded active active Slices
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of Temporary Directories

LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type.

79 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.

@.:/# systemctl enable udm-boot @.:/# systemctl status udm-boot

● udm-boot.service - Run On Startup UDM Loaded: loaded (/lib/systemd/system/udm-boot.service; enabled; vendor preset: enabled) Active: active (exited) since Sun 2023-01-29 11:19:34 AEDT; 1min 8s ago Main PID: 14314 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4736) Memory: 0B CPU: 0 CGroup: /system.slice/udm-boot.service

Jan 29 11:19:30 ubnt systemd[1]: Starting Run On Startup UDM... Jan 29 11:19:31 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/05-install-cni-plugins.sh Jan 29 11:19:32 ubnt ssh-proxy[14314]: Pouring /mnt/data/.cache/cni-plugins/cni-plugins-linux-arm64-v1.2.0.tgz Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/06-cni-bridge.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/99-run-vpn.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: [Sun Jan 29 11:19:34 AEDT 2023] split-vpn: tun1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:19:34 ubnt systemd[1]: Started Run On Startup UDM. @.***:/#

On Jan 25, 2023, at 3:14 AM, peacey @.***> wrote:

Great @david00769 https://github.com/david00769. Glad to see that fixed it. You should also set MSS_CLAMPING_IPV4="1382" or you might run into HTTPS sites not loading or stalling (keep this in mind if you don't change the option now).

For the boot script issue, if you already installed the boot service try this to fix it. There is a bug in udm-boot that doesn't enable it automatically. You have to do it manually the first time after installation. Run these commands:

unifi-os shell systemctl enable udm-boot systemctl status udm-boot exit Once you do that, make sure 99-run-vpn.sh is in /mnt/data/on_boot.d, and make sure it has executable permissions. Then restart and see if it runs at boot.

If it still doesn't run at boot, show me the output of the following after a reboot.

unifi-os shell systemctl status udm-boot — Reply to this email directly, view it on GitHub https://github.com/peacey/split-vpn/issues/156#issuecomment-1402209721, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5NTN2JR5NE2OPO6G4AVVQLWT75U5ANCNFSM6AAAAAAUEKOVDM. You are receiving this because you were mentioned.

[david00769]

@.***:/# systemctl cat udm-boot

/lib/systemd/system/udm-boot.service

[Unit] Description=Run On Startup UDM Wants=network-online.target After=network-online.target

[Service] Type=oneshot ExecStart=/sbin/ssh-proxy 'mkdir -p /mnt/data/on_boot.d && find -L /mnt/data/on_boot.d -iname ".sh" -mindepth 1 -maxdepth 1 -type f -print0 | sort -z | xargs -0 -r -n 1 -- sh -c '\''if test -x "$0"; then echo "%n: running $0"; "$0"; else case "$0" in .sh) echo "%n: sourcing $0"; . "$0";; *) echo "%n: ignoring $0";; esac; fi'\' RemainAfterExit=true

[Install] WantedBy=multi-user.target

@.:/# journalctl -xeu udm-boot -- Logs begin at Sun 2023-01-29 11:09:08 AEDT, end at Sun 2023-01-29 11:23:58 AEDT. -- Jan 29 11:09:09 ubnt systemd[1]: Starting Run On Startup UDM... -- Subject: A start job for unit udm-boot.service has begun execution -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit udm-boot.service has begun execution. -- -- The job identifier is 66. Jan 29 11:09:09 ubnt systemd[1]: udm-boot.service: Current command vanished from the unit file, execution of the command list won't be resumed. Jan 29 11:09:09 ubnt ssh-proxy[72]: udm-boot.service: running /mnt/data/on_boot.d/05-install-cni-plugins.sh Jan 29 11:09:09 ubnt ssh-proxy[72]: curl: (6) Could not resolve: github.com (Could not contact DNS servers) Jan 29 11:09:10 ubnt ssh-proxy[72]: Pouring /mnt/data/.cache/cni-plugins/cni-plugins-linux-arm64-latest.tgz Jan 29 11:09:13 ubnt ssh-proxy[72]: udm-boot.service: running /mnt/data/on_boot.d/06-cni-bridge.sh Jan 29 11:09:13 ubnt ssh-proxy[72]: udm-boot.service: running /mnt/data/on_boot.d/99-run-vpn.sh Jan 29 11:09:13 ubnt ssh-proxy[72]: [Sun Jan 29 11:09:13 AEDT 2023] split-vpn: tun1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:09:13 ubnt ssh-proxy[72]: Cannot find device "tun1" Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Main process exited, code=exited, status=123/n/a -- Subject: Unit process exited -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- An ExecStart= process belonging to unit udm-boot.service has exited. -- -- The process' exit code is 'exited' and its exit status is 123. Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- The unit udm-boot.service has entered the 'failed' state with result 'exit-code'. Jan 29 11:09:13 ubnt systemd[1]: Failed to start Run On Startup UDM. -- Subject: A start job for unit udm-boot.service has failed -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit udm-boot.service has finished with a failure. -- -- The job identifier is 66 and the job result is failed. Jan 29 11:09:13 ubnt systemd[1]: udm-boot.service: Consumed 101ms CPU time. -- Subject: Resources consumed by unit runtime -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- The unit udm-boot.service completed and consumed the indicated resources. Jan 29 11:19:30 ubnt systemd[1]: Starting Run On Startup UDM... -- Subject: A start job for unit udm-boot.service has begun execution -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit udm-boot.service has begun execution. -- -- The job identifier is 153. Jan 29 11:19:31 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/05-install-cni-plugins.sh Jan 29 11:19:32 ubnt ssh-proxy[14314]: Pouring /mnt/data/.cache/cni-plugins/cni-plugins-linux-arm64-v1.2.0.tgz Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/06-cni-bridge.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/99-run-vpn.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: [Sun Jan 29 11:19:34 AEDT 2023] split-vpn: tun1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:19:34 ubnt systemd[1]: Started Run On Startup UDM. -- Subject: A start job for unit udm-boot.service has finished successfully -- Defined-By: systemd -- Support: https://www.debian.org/support -- -- A start job for unit udm-boot.service has finished successfully. -- -- The job identifier is 153. @.:/#

On Jan 29, 2023, at 11:22 AM, David Siroky @.***> wrote:

I can get it up and running, but it does not survive a reboot

unifi-os shell

@.***:/# systemctl

UNIT LOAD ACTIVE SUB DESCRIPTION
dev-boot4.device loaded activating tentative /dev/boot4
dev-boot6.device loaded activating tentative /dev/boot6
dev-disk-sda1.device loaded activating tentative /dev/disk/sda1
-.mount loaded active mounted /
data.mount loaded active mounted /data
dev-disk.mount loaded active mounted /dev/disk
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
etc-hostname.mount loaded active mounted /etc/hostname
etc-hosts.mount loaded active mounted /etc/hosts
etc-resolv.conf.mount loaded active mounted /etc/resolv.conf
etc-systemd-system-unifi\x2dcore.service.d-capabilities\x2dworkaround.conf.mount loaded active mounted /etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf etc-unifi\x2dos-ssh_proxy_port.mount loaded active mounted /etc/unifi-os/ssh_proxy_port
etc_host.mount loaded active mounted /etc_host
mnt-persistent.mount loaded active mounted /mnt/persistent
root-.ssh-id_rsa.mount loaded active mounted /root/.ssh/id_rsa
run-.containerenv.mount loaded active mounted /run/.containerenv
srv.mount loaded active mounted /srv
sys-kernel-config.mount loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount loaded active mounted Kernel Debug File System
tmp.mount loaded active mounted /tmp
usr-lib-version.mount loaded active mounted /usr/lib/version
var-log-journal.mount loaded active mounted /var/log/journal
var-opt-unifi-tmp.mount loaded active mounted /var/opt/unifi/tmp
systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
init.scope loaded active running System and Service Manager
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
exim4.service loaded active running LSB: exim Mail Transport Agent
freeswitch.service loaded active running freeswitch
@. loaded active exited PostgreSQL initial setup service
postgresql.service loaded active exited PostgreSQL RDBMS
@. loaded active running PostgreSQL Cluster 9.6-main
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-modules-load.service loaded active exited Load Kernel Modules
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-sysusers.service loaded active exited Create System Users
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
● udm-boot.service loaded failed failed Run On Startup UDM

uid-agent.service loaded active running UID Agent service
ulp-go.service loaded active running ULP-GO
unifi-base-ucore.service loaded active running UniFi Base Controller
unifi-core.service loaded active running UniFi Core
unifi-pion-gw.service loaded active running Unifi Pion Gateway service
unifi.service loaded active running unifi
-.slice loaded active active Root Slice
system-postgresql.slice loaded active active system-postgresql.slice
system-postgresql\x2dcluster.slice loaded active active system-postgresql\x2dcluster.slice
system.slice loaded active active System Slice
user.slice loaded active active User and Session Slice
avahi-daemon.socket loaded active running Avahi mDNS/DNS-SD Stack Activation Socket
dbus.socket loaded active running D-Bus System Message Bus Socket
systemd-initctl.socket loaded active listening initctl Compatibility Named Pipe
systemd-journald-dev-log.socket loaded active running Journal Socket (/dev/log)
systemd-journald.socket loaded active running Journal Socket
basic.target loaded active active Basic System
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network.target loaded active active Network
paths.target loaded active active Paths
slices.target loaded active active Slices
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of Temporary Directories

LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type.

79 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'. @.:/# systemctl start udm-boot.service @.:/# systemctl UNIT LOAD ACTIVE SUB DESCRIPTION
dev-boot4.device loaded activating tentative /dev/boot4
dev-boot6.device loaded activating tentative /dev/boot6
dev-disk-sda1.device loaded activating tentative /dev/disk/sda1
-.mount loaded active mounted /
data.mount loaded active mounted /data
dev-disk.mount loaded active mounted /dev/disk
dev-hugepages.mount loaded active mounted Huge Pages File System
dev-mqueue.mount loaded active mounted POSIX Message Queue File System
etc-hostname.mount loaded active mounted /etc/hostname
etc-hosts.mount loaded active mounted /etc/hosts
etc-resolv.conf.mount loaded active mounted /etc/resolv.conf
etc-systemd-system-unifi\x2dcore.service.d-capabilities\x2dworkaround.conf.mount loaded active mounted /etc/systemd/system/unifi-core.service.d/capabilities-workaround.conf etc-unifi\x2dos-ssh_proxy_port.mount loaded active mounted /etc/unifi-os/ssh_proxy_port
etc_host.mount loaded active mounted /etc_host
mnt-persistent.mount loaded active mounted /mnt/persistent
root-.ssh-id_rsa.mount loaded active mounted /root/.ssh/id_rsa
run-.containerenv.mount loaded active mounted /run/.containerenv
srv.mount loaded active mounted /srv
sys-kernel-config.mount loaded active mounted Kernel Configuration File System
sys-kernel-debug.mount loaded active mounted Kernel Debug File System
tmp.mount loaded active mounted /tmp
usr-lib-version.mount loaded active mounted /usr/lib/version
var-log-journal.mount loaded active mounted /var/log/journal
var-opt-unifi-tmp.mount loaded active mounted /var/opt/unifi/tmp
systemd-ask-password-console.path loaded active waiting Dispatch Password Requests to Console Directory Watch
systemd-ask-password-wall.path loaded active waiting Forward Password Requests to Wall Directory Watch
init.scope loaded active running System and Service Manager
avahi-daemon.service loaded active running Avahi mDNS/DNS-SD Stack
cron.service loaded active running Regular background program processing daemon
dbus.service loaded active running D-Bus System Message Bus
exim4.service loaded active running LSB: exim Mail Transport Agent
freeswitch.service loaded active running freeswitch
@. loaded active exited PostgreSQL initial setup service
postgresql.service loaded active exited PostgreSQL RDBMS
@. loaded active running PostgreSQL Cluster 9.6-main
systemd-journal-flush.service loaded active exited Flush Journal to Persistent Storage
systemd-journald.service loaded active running Journal Service
systemd-logind.service loaded active running Login Service
systemd-modules-load.service loaded active exited Load Kernel Modules
systemd-remount-fs.service loaded active exited Remount Root and Kernel File Systems
systemd-sysctl.service loaded active exited Apply Kernel Variables
systemd-sysusers.service loaded active exited Create System Users
systemd-tmpfiles-setup-dev.service loaded active exited Create Static Device Nodes in /dev
systemd-tmpfiles-setup.service loaded active exited Create Volatile Files and Directories
systemd-update-utmp.service loaded active exited Update UTMP about System Boot/Shutdown
systemd-user-sessions.service loaded active exited Permit User Sessions
udm-boot.service loaded active exited Run On Startup UDM
uid-agent.service loaded active running UID Agent service
ulp-go.service loaded active running ULP-GO
unifi-base-ucore.service loaded active running UniFi Base Controller
unifi-core.service loaded active running UniFi Core
unifi-pion-gw.service loaded active running Unifi Pion Gateway service
unifi.service loaded active running unifi
-.slice loaded active active Root Slice
system-postgresql.slice loaded active active system-postgresql.slice
system-postgresql\x2dcluster.slice loaded active active system-postgresql\x2dcluster.slice
system.slice loaded active active System Slice
user.slice loaded active active User and Session Slice
avahi-daemon.socket loaded active running Avahi mDNS/DNS-SD Stack Activation Socket
dbus.socket loaded active running D-Bus System Message Bus Socket
systemd-initctl.socket loaded active listening initctl Compatibility Named Pipe
systemd-journald-dev-log.socket loaded active running Journal Socket (/dev/log)
systemd-journald.socket loaded active running Journal Socket
basic.target loaded active active Basic System
cryptsetup.target loaded active active Local Encrypted Volumes
getty.target loaded active active Login Prompts
graphical.target loaded active active Graphical Interface
local-fs-pre.target loaded active active Local File Systems (Pre)
local-fs.target loaded active active Local File Systems
multi-user.target loaded active active Multi-User System
network-online.target loaded active active Network is Online
network.target loaded active active Network
paths.target loaded active active Paths
slices.target loaded active active Slices
sockets.target loaded active active Sockets
swap.target loaded active active Swap
sysinit.target loaded active active System Initialization
timers.target loaded active active Timers
systemd-tmpfiles-clean.timer loaded active waiting Daily Cleanup of Temporary Directories

LOAD = Reflects whether the unit definition was properly loaded. ACTIVE = The high-level unit activation state, i.e. generalization of SUB. SUB = The low-level unit activation state, values depend on unit type.

79 loaded units listed. Pass --all to see loaded but inactive units, too. To show all installed unit files use 'systemctl list-unit-files'.

@.:/# systemctl enable udm-boot @.:/# systemctl status udm-boot

● udm-boot.service - Run On Startup UDM Loaded: loaded (/lib/systemd/system/udm-boot.service; enabled; vendor preset: enabled) Active: active (exited) since Sun 2023-01-29 11:19:34 AEDT; 1min 8s ago Main PID: 14314 (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4736) Memory: 0B CPU: 0 CGroup: /system.slice/udm-boot.service

Jan 29 11:19:30 ubnt systemd[1]: Starting Run On Startup UDM... Jan 29 11:19:31 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/05-install-cni-plugins.sh Jan 29 11:19:32 ubnt ssh-proxy[14314]: Pouring /mnt/data/.cache/cni-plugins/cni-plugins-linux-arm64-v1.2.0.tgz Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/06-cni-bridge.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: udm-boot.service: running /mnt/data/on_boot.d/99-run-vpn.sh Jan 29 11:19:34 ubnt ssh-proxy[14314]: [Sun Jan 29 11:19:34 AEDT 2023] split-vpn: tun1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:19:34 ubnt systemd[1]: Started Run On Startup UDM. @.***:/#

On Jan 25, 2023, at 3:14 AM, peacey @.***> wrote:

Great @david00769 https://github.com/david00769. Glad to see that fixed it. You should also set MSS_CLAMPING_IPV4="1382" or you might run into HTTPS sites not loading or stalling (keep this in mind if you don't change the option now).

For the boot script issue, if you already installed the boot service try this to fix it. There is a bug in udm-boot that doesn't enable it automatically. You have to do it manually the first time after installation. Run these commands:

unifi-os shell systemctl enable udm-boot systemctl status udm-boot exit Once you do that, make sure 99-run-vpn.sh is in /mnt/data/on_boot.d, and make sure it has executable permissions. Then restart and see if it runs at boot.

If it still doesn't run at boot, show me the output of the following after a reboot.

unifi-os shell systemctl status udm-boot — Reply to this email directly, view it on GitHub https://github.com/peacey/split-vpn/issues/156#issuecomment-1402209721, or unsubscribe https://github.com/notifications/unsubscribe-auth/A5NTN2JR5NE2OPO6G4AVVQLWT75U5ANCNFSM6AAAAAAUEKOVDM. You are receiving this because you were mentioned.

[peacey]

Hey @david00769,

From your log nothing is wrong with the boot service, it's running fine. The problem is the site-to-site interface tun1 isn't created till after this script runs, so the script is failing on that device not being found:

Jan 29 11:09:13 ubnt ssh-proxy[72]: [Sun Jan 29 11:09:13 AEDT 2023] split-vpn: t un1 up: Loading configuration from /etc/split-vpn/nexthop/site1/vpn.conf. Jan 29 11:09:13 ubnt ssh-proxy[72]: Cannot find device "tun1"

So what you should do is wait for tun1 to come up before running the up command. Just modify your run-vpn.sh in your nexthop folder and replace it with:

#!/bin/sh

# Load configuration and bring routes up
cd /etc/split-vpn/nexthop/target
. ./vpn.conf

while ! ip link show dev ${DEV} 1>/dev/null 2>&1; do
     sleep 5
done

/etc/split-vpn/vpn/updown.sh ${DEV} up site1

Make sure to replace the cd command near the top with the correct directory, and the site1 at the bottom with whatever nickname you were using.

That while loop should make it check for tun1 every 5 seconds until the site-to-site device is up, then it will continue with the updown command. See if that works for you.

[david00769]

Thanks - that’s what I’d concluded also - it was erroring out because the site to site vpn wasn’t yet established. Later when I ran it manually it succeeded because the tunnel was up.

Will try add the sleep timer and see how we go!

On Jan 29, 2023, at 12:01 PM, peacey @.***> wrote:

while ! ip link show dev ${DEV} 1>/dev/null 2>&1; do sleep 5 done