search

peacey / split-vpn

A split tunnel VPN script for Unifi OS routers (UDM, UXG, UDR) with policy based routing.
GNU General Public License v3.0
802 stars 56 forks source link

Split-vpn Wirequard access to S2S VPN query #172

Open sudo-kraken opened 1 year ago

sudo-kraken commented 1 year ago

I have wireguard-kmod setup up and working I can access all VLANs I need to from the client however I have a S2S VPN setup in the UDM Pro to connect to my friends network, which is reachable from my VLANs but my Wireguard VPN can not see anything across the S2S VPN, how do I fix this as I need access to the other site, meaning that it would go Client >> My UDMP >> S2S Endpoint. I assume this is a routing issue but I cant fathom it out.

is it possible to use these scripts to have S2S endpoints available over the wireguard vpn using split-vpn? current config below:

Server

[Interface]
Address = 10.10.10.1/24
ListenPort = 51820
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
AllowedIPs = 10.10.10.2/32

Client

[Interface]
Address = 10.10.10.2/24
PrivateKey = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
ListenPort = 21841

[Peer]
PublicKey = xxxxxxxxxxxxxxxxxxxxxxxxxx
Endpoint = xx.xx.xxx.xx:51820
AllowedIPs = 10.10.10.0/24,192.168.100.0/24

PersistentKeepalive = 25

This is the configuration that I have. I can't access the subnet 192.168.100.0/24 which is a site to site ipsec-vpn configured in the UDM Pro, if I change this to be any of my other local subnets then access works flawlessly, I just cant route over to the remote subnet on the S2S VPN.

peacey commented 1 year ago

Hi @Joeharrison94,

Sorry, it took me a while but I think I understand the setup. So basically it's like this:

WG Client -> wireguard -> UDM (WG server) -> S2S -> UDM2

In that case, you don't need to use split-vpn, you're just missing a route on UDM2 (your friend's UDM). Basically, your UDM knows that 192.168.1.100/24 should be forwarded through the S2S to UDM2, but UDM2 doesn't know that 10.10.10.0/24 subnet should be routed back to your UDM. So in your S2S config on your friend's UDM (not your UDM), you need to add the WG subnet (10.10.10.0/24) to the "Remote Subnets" list. That should hopefully fix it.

sudo-kraken commented 1 year ago

That's the problem it's added on his side to the remote subnets list yet I can't get to him at all.

peacey commented 1 year ago

Are you using IPsec or openvpn site-to-site? Do you have only one site-to-site setup? Can you show me the output of ip link on the UDM so we can see the interface name for the S2S? Also please show me the output of ip route on the UDM.

Can you run the following tcpdump to an IP on 192.168.100.0/24 that you can ping normally without wireguard? Like for example 192.168.100.3 or something else. Then run this on the UDMP

tcpdump -ni any host 192.168.100.3

Then ping 192.168.100.3 from a Wireguard client and show me the tcpdump output.

Then run this tcpdump on the UDM (if you use IPsec use vti64, but if you use openvpn S2S, then use tun0 instead of vti64 in the tcpdump below. 

tcpdump -ni vti64 host 192.168.100.3

Then again ping 192.168.100.3 from a wireguard client and show me the output of the tcpdump.

Make sure to change 192.169.100.3 in all above commands if you're pinging a different IP. Make sure it exists and you can ping it from a normal UDM client.

sudo-kraken commented 1 year ago

We use IPsec for the S2S.

ip link show:

ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 76:ac:b9:1c:92:a1 brd 00:00:00:00:00:00
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 36:94:f9:c5:2d:43 brd ff:ff:ff:ff:ff:ff
3: eth9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 10000
    link/ether 74:ac:b9:1c:92:aa brd ff:ff:ff:ff:ff:ff
4: eth8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a9 brd ff:ff:ff:ff:ff:ff
5: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master br0 state UP mode DEFAULT group default qlen 10000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
6: switch0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
7: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
11: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
12: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
    link/ether 7a:e9:24:cf:38:0c brd ff:ff:ff:ff:ff:ff
13: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
    link/ether 32:53:b3:52:78:61 brd ff:ff:ff:ff:ff:ff
14: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
15: eth0@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a1 brd ff:ff:ff:ff:ff:ff
16: eth1@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
17: eth2@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a3 brd ff:ff:ff:ff:ff:ff
18: eth3@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a4 brd ff:ff:ff:ff:ff:ff
19: eth4@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a5 brd ff:ff:ff:ff:ff:ff
20: eth5@switch0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a6 brd ff:ff:ff:ff:ff:ff
21: eth6@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a7 brd ff:ff:ff:ff:ff:ff
22: eth7@switch0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a8 brd ff:ff:ff:ff:ff:ff
23: eth10.110@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br110 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
24: eth10.140@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br140 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
25: eth10.150@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br150 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
26: eth10.160@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br160 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
27: eth10.170@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br170 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
29: switch0.1@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
30: switch0.110@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br110 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
31: switch0.140@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br140 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
32: switch0.150@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br150 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
33: switch0.160@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br160 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
34: switch0.170@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br170 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
36: br110: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
37: br140: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
38: br150: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
39: br160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
40: br170: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
52: cni0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:02:c1:31:c4:b2 brd ff:ff:ff:ff:ff:ff
55: dnsfilter: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 66:e9:b5:2d:0d:18 brd ff:ff:ff:ff:ff:ff
57: dnsfilter-0@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dnsfilter state UP mode DEFAULT group default qlen 1000
    link/ether 66:e9:b5:2d:0d:18 brd ff:ff:ff:ff:ff:ff link-netns dns-10.0.160.1
59: dnsfilter-1@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dnsfilter state UP mode DEFAULT group default qlen 1000
    link/ether ce:b9:3d:c0:28:d1 brd ff:ff:ff:ff:ff:ff link-netns dns-10.0.150.1
60: honeypot0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 62:83:a4:d1:e3:8a brd ff:ff:ff:ff:ff:ff
61: honeypot110: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ea:ba:fe:e4:c0:7e brd ff:ff:ff:ff:ff:ff
62: honeypot140: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1e:7a:3a:d1:76:d2 brd ff:ff:ff:ff:ff:ff
63: honeypot150: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 16:22:a2:7c:ef:1c brd ff:ff:ff:ff:ff:ff
64: honeypot160: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 46:ac:7e:86:74:8d brd ff:ff:ff:ff:ff:ff
65: honeypot170: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether e6:98:95:83:36:8b brd ff:ff:ff:ff:ff:ff
root@Harrison-UDM-Pro:~# ip sh  ^C
root@Harrison-UDM-Pro:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 76:ac:b9:1c:92:a1 brd 00:00:00:00:00:00
2: dummy0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 36:94:f9:c5:2d:43 brd ff:ff:ff:ff:ff:ff
3: eth9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 10000
    link/ether 74:ac:b9:1c:92:aa brd ff:ff:ff:ff:ff:ff
4: eth8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a9 brd ff:ff:ff:ff:ff:ff
5: eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc mq master br0 state UP mode DEFAULT group default qlen 10000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
6: switch0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9216 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
7: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
8: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
9: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
10: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
11: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
12: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
    link/ether 7a:e9:24:cf:38:0c brd ff:ff:ff:ff:ff:ff
13: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 32
    link/ether 32:53:b3:52:78:61 brd ff:ff:ff:ff:ff:ff
14: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
15: eth0@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a1 brd ff:ff:ff:ff:ff:ff
16: eth1@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
17: eth2@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a3 brd ff:ff:ff:ff:ff:ff
18: eth3@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a4 brd ff:ff:ff:ff:ff:ff
19: eth4@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a5 brd ff:ff:ff:ff:ff:ff
20: eth5@switch0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a6 brd ff:ff:ff:ff:ff:ff
21: eth6@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a7 brd ff:ff:ff:ff:ff:ff
22: eth7@switch0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:a8 brd ff:ff:ff:ff:ff:ff
23: eth10.110@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br110 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
24: eth10.140@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br140 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
25: eth10.150@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br150 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
26: eth10.160@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br160 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
27: eth10.170@eth10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br170 state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
29: switch0.1@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
30: switch0.110@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br110 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
31: switch0.140@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br140 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
32: switch0.150@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br150 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
33: switch0.160@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br160 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
34: switch0.170@switch0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue master br170 state UP mode DEFAULT group default qlen 1000
    link/ether 76:ac:b9:1c:92:a2 brd ff:ff:ff:ff:ff:ff
36: br110: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
37: br140: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
38: br150: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
39: br160: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
40: br170: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9216 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 74:ac:b9:1c:92:ab brd ff:ff:ff:ff:ff:ff
52: cni0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether c2:02:c1:31:c4:b2 brd ff:ff:ff:ff:ff:ff
55: dnsfilter: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 66:e9:b5:2d:0d:18 brd ff:ff:ff:ff:ff:ff
57: dnsfilter-0@if56: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dnsfilter state UP mode DEFAULT group default qlen 1000
    link/ether 66:e9:b5:2d:0d:18 brd ff:ff:ff:ff:ff:ff link-netns dns-10.0.160.1
59: dnsfilter-1@if58: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master dnsfilter state UP mode DEFAULT group default qlen 1000
    link/ether ce:b9:3d:c0:28:d1 brd ff:ff:ff:ff:ff:ff link-netns dns-10.0.150.1
60: honeypot0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 62:83:a4:d1:e3:8a brd ff:ff:ff:ff:ff:ff
61: honeypot110: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ea:ba:fe:e4:c0:7e brd ff:ff:ff:ff:ff:ff
62: honeypot140: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 1e:7a:3a:d1:76:d2 brd ff:ff:ff:ff:ff:ff
63: honeypot150: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 16:22:a2:7c:ef:1c brd ff:ff:ff:ff:ff:ff
64: honeypot160: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 46:ac:7e:86:74:8d brd ff:ff:ff:ff:ff:ff
65: honeypot170: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether e6:98:95:83:36:8b brd ff:ff:ff:ff:ff:ff

TCP dump:

tcpdump -ni any host 192.168.100.21
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
11:12:35.002496 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 3599553727:3599553759, ack 2381489489, win 1021, length 32
11:12:35.002620 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 0:32, ack 1, win 1021, length 32
11:12:35.002631 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 0:32, ack 1, win 1021, length 32
11:12:35.003679 ethertype IPv4, IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1403, length 36
11:12:35.003679 IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1403, length 36
11:12:35.003679 IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1403, length 36
11:12:35.048994 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 3275085406:3275085438, ack 3595193817, win 1021, length 32
11:12:35.049164 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 0:32, ack 1, win 1021, length 32
11:12:35.049178 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 0:32, ack 1, win 1021, length 32
11:12:35.049954 ethertype IPv4, IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1636, length 36
11:12:35.049954 IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1636, length 36
11:12:35.049954 IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 1:37, ack 32, win 1636, length 36
11:12:35.062982 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 37, win 1021, length 0
11:12:35.063115 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 37, win 1021, length 0
11:12:35.063127 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 37, win 1021, length 0
11:12:35.109189 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 37, win 1021, length 0
11:12:35.109316 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 37, win 1021, length 0
11:12:35.109328 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 37, win 1021, length 0
11:12:39.857552 IP 10.10.10.2 > 192.168.100.21: ICMP echo request, id 275, seq 1, length 64
11:12:39.857615 IP 82.34.XX.XX > 192.168.100.21: ICMP echo request, id 275, seq 1, length 64
11:12:43.866566 IP 10.10.10.2 > 192.168.100.21: ICMP echo request, id 362, seq 1, length 64
11:12:43.866640 IP 82.34.XX.XX > 192.168.100.21: ICMP echo request, id 362, seq 1, length 64
11:12:45.003525 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.003654 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.003665 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.004620 ethertype IPv4, IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1403, length 36
11:12:45.004620 IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1403, length 36
11:12:45.004620 IP 10.0.160.40.53256 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1403, length 36
11:12:45.049327 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.049516 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.049542 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [P.], seq 32:64, ack 37, win 1021, length 32
11:12:45.050468 ethertype IPv4, IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1636, length 36
11:12:45.050468 IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1636, length 36
11:12:45.050468 IP 10.0.160.40.53258 > 192.168.100.21.32400: Flags [P.], seq 37:73, ack 64, win 1636, length 36
11:12:45.064864 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 73, win 1021, length 0
11:12:45.065038 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 73, win 1021, length 0
11:12:45.065053 IP 192.168.100.21.32400 > 10.0.160.40.53256: Flags [.], ack 73, win 1021, length 0
11:12:45.109023 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 73, win 1021, length 0
11:12:45.109170 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 73, win 1021, length 0
11:12:45.109184 IP 192.168.100.21.32400 > 10.0.160.40.53258: Flags [.], ack 73, win 1021, length 0
^C
40 packets captured
40 packets received by filter
0 packets dropped by kernel

this line is the ping from my phone on the wg trying to connect to it:

11:12:39.857552 IP 10.10.10.2 > 192.168.100.21: ICMP echo request, id 275, seq 1, length 64

the only IP I can use on the S2S is 192.168.100.21 as this is a server

peacey commented 1 year ago

That's odd, I don't see any IPSec interface in your ip link output. Are you sure the IPSec is connected?

Can you show me the output of ip route so we can see your route table?

Also, I see this line when you try to ping:

11:12:39.857552 IP 10.10.10.2 > 192.168.100.21: ICMP echo request, id 275, seq 1, length 64
11:12:39.857615 IP 82.34.XX.XX > 192.168.100.21: ICMP echo request, id 275, seq 1, length 64

Is 82.34.XX.XX your WAN IP? So the requests to 192.168.100.21 from your WG client seem to be going out your WAN instead of the IPSec tunnel (so other side isn't receiving them). That shouldn't happen if the routes are setup, but again I couldn't see the IPSec interface in your interface list.

sudo-kraken commented 1 year ago

Yes thats the WAN interface and the IPsec tunnel is 100% working I have access from all my VLANs my side.

IP route output:

ip route
10.0.100.0/24 dev br0 proto kernel scope link src 10.0.100.1
10.0.110.0/24 dev br110 proto kernel scope link src 10.0.110.1
10.0.140.0/24 dev br140 proto kernel scope link src 10.0.140.1
10.0.150.0/24 dev br150 proto kernel scope link src 10.0.150.1
10.0.160.0/24 dev br160 proto kernel scope link src 10.0.160.1
10.0.170.0/24 dev br170 proto kernel scope link src 10.0.170.1
10.10.10.0/24 dev wg0 proto kernel scope link src 10.10.10.1
82.34.XX.XX/20 dev eth8 proto kernel scope link src 82.34.XX.XX
203.0.113.0/24 dev dnsfilter proto kernel scope link src 203.0.113.1
peacey commented 1 year ago

Thanks @Joeharrison94. Again really weird, I don't see the routes to your IPsec tunnel (there should be a route entry for 192.168.100.0/24). I've never seen an IPSec tunnel on the UDM set up this way without a route or IPsec interface present, so it's really confusing. In your Unifi S2S settings, is "Route based VPN" checked, or are you using a policy based?

Can you show me the output of ip rule?

sudo-kraken commented 1 year ago

IP Rule output:

0:      from all lookup local
220:    from all lookup 220
32000:  from all lookup main
32500:  from 82.34XX.XX lookup 201
32766:  from all lookup 201
32767:  from all lookup default

This is the settings of the IPsec tunnel image

peacey commented 1 year ago

Oh, you don't have Route based VPN checked in your settings. That explains why the interface and routes aren't there. I don't believe you can forward custom traffic through the IPSec without it being route based, because the configuration for what subnets should be allowed through the tunnel is done in the StrongSwan IPSec config rather than by a route table.

Is there a reason you're not using a route based VPN? Unifi uses route-based by default, so I'm wondering why you decided to uncheck it?

sudo-kraken commented 1 year ago

It defaulted to non-route based, if I enable this does it create the routes itself?

sudo-kraken commented 1 year ago

What do I need to do on my setup and my friends to get route based working in the same way as it is now

peacey commented 1 year ago

Yes it should, but you need to enable Route-based on both sides of the S2S settings.

Then once you do, you can verify the route is there with ip route.

sudo-kraken commented 1 year ago

I will update it and let you know the result of the route based ones, I have added my config just waiting on my friend, I can see the route in the route table for this subnet now.

peacey commented 1 year ago

Great @Joeharrison94! Once you enable it on your friend's, make sure you can still ping 192.168.100.21 from a local non-WG client so we see everything is working normally.

Also, pinging from a WG client should start working hopefully, so test that after too.

sudo-kraken commented 1 year ago

What is weird is that he hasnt made the changes yet so I cant get to him from my network, but the wireguard client can get to it already.

peacey commented 1 year ago

That's interesting. Maybe you already have route-based checked on his side, or maybe the policy-based is stateful so it knows to send the reply packet back out the IPSec tunnel if the request came from the tunnel... Not sure. But better both sides be the same in any case, since that's what Ubiquiti says to do.

But I'm glad pinging from a WG client worked! Hopefully that means your problem is fixed once you confirm his settings.

sudo-kraken commented 1 year ago

So when he enabled the options his side to match mine, everything stopped working.

sudo-kraken commented 1 year ago 
vti64: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1419 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ipip 82.xx.xx.xx peer 92.xx.xx.xx
192.168.100.0/24 dev vti64 proto static scope link metric 30
peacey commented 1 year ago

That's weird, @Joeharrison94. It should work fine. What is not working? Are you saying you cannot ping 192.168.100.21 from a local LAN client anymore (non-WG)? Did you make sure his settings has Route-based checked now, and does he have your LAN subnet added to his remote subnets list?

Have you tried to turn off the IPSec tunnel and back on?

sudo-kraken commented 1 year ago

Yeah, confirmed settings on both sides all subnets listed correctly, with route based disabled my LAN works and can get to everything it needs to, when enabled on my side not his my wg client can get there but not my LAN, when both sides enabled nothing works at all.

peacey commented 1 year ago

Odd, it should just work. Even in the Unifi IPSec guide, it says they use Route-based by default, so there shouldn't be anything extra to do. Just double check all the parameters are the same, including security options like DH group and IKE version. Then try to restart both tunnels.

Does he have any firewall rules that can block access?

We can check to see that packets from your router is making it out the IPSec tunnel by doing a tcpdump like last time and pinging from a local client, but to really debug we need to do a tcpdump on his end too. But that's a hassle since you don't have access to his router.

Can you just run the tcpdump and do a ping, and also can you run this command?

ipsec statusall