unifi-utilities / unifios-utilities

A collection of enhancements for UnifiOS based devices
GNU General Public License v3.0
3.86k stars 415 forks source link

Wireguard almost works #211

Closed turowicz closed 2 years ago

turowicz commented 3 years ago

Hi @boostchicken,

First I'd like to thank you for your amazing work on this repository!

I have followed the guide here: https://github.com/boostchicken/udm-utilities/wiki/Run-a-Wireguard-VPN-server-on-UDM-Pro

The wg0 connection does indeed come up and I do complete a handshake, but shortly after the connection fails. I am unable to access any resources through the VPN.

Client Logs:

[  +7,696954] wireguard: wg0: Interface created
[  +0,043608] wireguard: wg0: Peer 75 created
[  +0,047732] wireguard: wg0: Sending keepalive packet to peer 75 (217.97.99.123:51820)
[  +0,000038] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)
[  +0,007134] wireguard: wg0: Receiving handshake response from peer 75 (217.97.99.123:51820)
[  +0,000008] wireguard: wg0: Keypair 26 created for peer 75
[lip19 16:44] wireguard: wg0: Retrying handshake with peer 75 (217.97.99.123:51820) because we stopped hearing back after 15 seconds
[  +0,000090] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)
[  +5,117206] wireguard: wg0: Handshake for peer 75 (217.97.99.123:51820) did not complete after 5 seconds, retrying (try 2)
[  +0,000164] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)
[  +5,377042] wireguard: wg0: Handshake for peer 75 (217.97.99.123:51820) did not complete after 5 seconds, retrying (try 3)
[  +0,000027] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)
[  +4,857534] wireguard: wg0: Retrying handshake with peer 75 (217.97.99.123:51820) because we stopped hearing back after 15 seconds
[  +0,511715] wireguard: wg0: Handshake for peer 75 (217.97.99.123:51820) did not complete after 5 seconds, retrying (try 2)
[  +0,000059] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)
[  +5,373881] wireguard: wg0: Handshake for peer 75 (217.97.99.123:51820) did not complete after 5 seconds, retrying (try 3)
[  +0,000064] wireguard: wg0: Sending handshake initiation to peer 75 (217.97.99.123:51820)

Server Logs:

# podman logs -f wireguard
[#] ip link add wg0 type wireguard
RTNETLINK answers: Not supported
[!] Missing WireGuard kernel module. Falling back to slow userspace implementation.
[#] wireguard-go wg0
┌──────────────────────────────────────────────────────┐
│                                                      │
│   Running wireguard-go is not required because this  │
│   kernel has first class support for WireGuard. For  │
│   information on installing the kernel module,       │
│   please visit:                                      │
│         https://www.wireguard.com/install/           │
│                                                      │
└──────────────────────────────────────────────────────┘
DEBUG: (wg0) 2021/07/19 14:43:19 Starting wireguard-go version 0.0.20210424
[#] wg setconf wg0 /dev/fd/63
Warning: AllowedIP has nonzero host part: 192.168.101.2/24
DEBUG: (wg0) 2021/07/19 14:43:19 Starting wireguard-go version 0.0.20210424
DEBUG: (wg0) 2021/07/19 14:43:19 Device started
DEBUG: (wg0) 2021/07/19 14:43:19 UAPI listener started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: handshake worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: TUN reader - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: event worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Interface up requested
DEBUG: (wg0) 2021/07/19 14:43:19 UDP bind has been updated
DEBUG: (wg0) 2021/07/19 14:43:19 Interface state was Down, requested Up, now Up
DEBUG: (wg0) 2021/07/19 14:43:19 UAPI: Updating private key
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v6 - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: decryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: encryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: decryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: handshake worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: encryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: decryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: handshake worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: encryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: decryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: handshake worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: encryption worker - started
DEBUG: (wg0) 2021/07/19 14:43:19 UAPI: Updating listen port
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v4 - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v6 - stopped
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v4 - stopped
DEBUG: (wg0) 2021/07/19 14:43:19 UDP bind has been updated
DEBUG: (wg0) 2021/07/19 14:43:19 UAPI: Updating fwmark
DEBUG: (wg0) 2021/07/19 14:43:19 UAPI: Removing all peers
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v4 - started
DEBUG: (wg0) 2021/07/19 14:43:19 Routine: receive incoming v6 - started
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - Starting...
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - UAPI: Created
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - UAPI: Removing all allowedips
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - UAPI: Adding allowedip
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - Routine: sequential receiver - started
DEBUG: (wg0) 2021/07/19 14:43:19 peer(BymT…cyjs) - Routine: sequential sender - started
[#] ip -4 address add 192.168.101.1/24 dev wg0
[#] ip link set mtu 1420 up dev wg0

DEBUG: (wg0) 2021/07/19 14:44:16 peer(BymT…cyjs) - Received handshake initiation
DEBUG: (wg0) 2021/07/19 14:44:16 peer(BymT…cyjs) - Sending handshake response
DEBUG: (wg0) 2021/07/19 14:44:16 peer(BymT…cyjs) - Receiving keepalive packet

My confguration:

VPN network on 192.168.101.0/24 VPN server on 192.168.101.1/24 VPN port 51820 Private Networks: 192.168.100.0/24, 192.168.101.0/24, 192.168.102.0/24, 192.168.200.0/24

All rules configured as you mention, except I am using different IP for the VPN network, 101 instead 240.

turowicz commented 3 years ago

My understanding is that after establishing the connection and routing everything through VPN (AllowedIPs = 0.0.0.0/0) it no longer works.

turowicz commented 3 years ago

Server config:

[Interface]
Address = 192.168.101.1/24
ListenPort = 51820
PrivateKey = 0MCG...XG0=
#PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o eth8 -j MASQUERADE
#PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o eth8 -j MASQUERADE

[Peer]
PublicKey = BymT...yjs=
AllowedIPs = 192.168.101.2/24
turowicz commented 3 years ago

Client config:

[Interface]
Address = 192.168.101.2/24
PrivateKey = iPH...8VQ=
DNS = 192.168.101.1

[Peer]
PublicKey = LIe...B1E=
Endpoint = [redacted]:51820
AllowedIPs = 0.0.0.0/0
PersistentKeepalive = 21
turowicz commented 3 years ago

After replacing 0.0.0.0/0 on the client with the private networks AllowedIPs = 192.168.100.0/24, 192.168.101.0/24, 192.168.102.0/24, 192.168.200.0/24 it no longer drops and I get:

[  +4,210753] wireguard: wg0: Receiving keepalive packet from peer 78 (217.97.99.123:51820)
[  +9,984748] wireguard: wg0: Receiving keepalive packet from peer 78 (217.97.99.123:51820)
[lip19 17:12] wireguard: wg0: Receiving keepalive packet from peer 78 (217.97.99.123:51820)
[ +10,042259] wireguard: wg0: Receiving keepalive packet from peer 78 (217.97.99.123:51820)

But I can't access any LAN resource.

turowicz commented 3 years ago

The following changes have helped and now I connect to LAN resources:

turowicz commented 3 years ago

Unfortunately the internet works terribly

turowicz commented 3 years ago

OK it now works all right with the following:

Server:

[Interface]
ListenPort = 51820      
PrivateKey = 0MC...XG0=

[Peer]                                                                                                
PublicKey = Bym...yjs=                                              
AllowedIPs = 192.168.101.2/32

Client

[Interface]
PrivateKey = iPH...8VQ=
Address = 192.168.101.2/32
DNS = 192.168.101.1

[Peer]
PublicKey = LIe...B1E=
AllowedIPs = 192.168.0.0/16
Endpoint = [redacted]:51820
PersistentKeepalive = 21
turowicz commented 3 years ago

Unfortunately for some reason the internet is unreliable,

Logs keep saying:

[  +1,687070] IPv6: ADDRCONF(NETDEV_CHANGE): calid015d1cc1f0: link becomes ready
[  +0,000032] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,065509] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,057192] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,713200] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,144245] nvidia-nvswitch: Version mismatch, kernel version 465.31 user version 450.51.06
[  +2,199836] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +8,346095] IPv6: ADDRCONF(NETDEV_CHANGE): calie2ee7c43ff1: link becomes ready
[  +0,000023] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,376580] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +1,013473] IPv6: ADDRCONF(NETDEV_CHANGE): cali89b7d9559c1: link becomes ready
[  +0,000021] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[  +0,051982] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

As if the eth0 was having trouble

turowicz commented 3 years ago

Although the public internet traffic doesn't go through wg0, when wg0 is up, it somehow impacts the public internet connectivity. It becomes really flaky. When I go to google.com and ask for my IP it does give me the local public internet IP, not the server's.

Why would this have an impact?

gtrabanco commented 2 years ago

Try with this:

Server

[Interface]
# Name = Wireguard Server
Address = 192.168.101.1/24
ListenPort = 51820
PrivateKey = #Server Private Key#
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostUp = echo "$(date +%s) WireGuard Started" >> /var/log/wireguard.log
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE
PostDown = echo "$(date +%s) WireGuard Going Down" >> /var/log/wireguard.log

[Peer]
# Name = Peer 1
PublicKey = #Client Public Key#
AllowedIPs = 192.168.101.2/32

Client

[Interface]
# Name = Peer 1
Address = 192.168.101.2/32
PrivateKey = # Client Private Key #
DNS = 1.1.1.1,1.0.0.1

[Peer]
# Name = Wireguard Server
Endpoint = #Server FQDN or IP with :51820
PublicKey = # Server Public Key #
# AllowedIPs = 0.0.0.0/1, 128.0.0.0/1, ::/1, 8000::/1 # This is necessary if you have some advanced routing, check split-vpn project.
AllowedIPs = 0.0.0.0/0, ::/0

With my knowledge could be:

If you do not want to tunnel traffic between your peer and wg to go to Internet and route only local traffic see my repository wireguard-site-to-site.

If you any doubt with wg configuration check pirate docs

alxwolf commented 2 years ago

Any progress on this topic @turowicz ?

With the simple files described in Step 4 and 5 of the wiki it should work. If so, maybe you can close the issue.

turowicz commented 2 years ago

Hey @alxwolf I managed to solve it by changing the DNS to:

DNS = 1.1.1.1,1.0.0.1,192.168.101.1

All other files as per my previous https://github.com/boostchicken/udm-utilities/issues/211#issuecomment-883505518 untouched.

Changing the files as you suggested resulted in failed connections.

It all works perfectly now. Seems like the public internet DNS just needs to be in front of the VPN one.

turowicz commented 2 years ago

*Actually I've also changed the AllowedIPs to my VPN and on-premise networks on UDM.

AllowedIPs = 192.168.100.0/24, 192.168.101.0/24, 192.168.200.0/24
up2geek commented 2 years ago

OK it now works all right with the following:

Server:

[Interface]
ListenPort = 51820      
PrivateKey = 0MC...XG0=

[Peer]                                                                                                
PublicKey = Bym...yjs=                                              
AllowedIPs = 192.168.101.2/32

Client

[Interface]
PrivateKey = iPH...8VQ=
Address = 192.168.101.2/32
DNS = 192.168.101.1

[Peer]
PublicKey = LIe...B1E=
AllowedIPs = 192.168.0.0/16
Endpoint = [redacted]:51820
PersistentKeepalive = 21

I logged in just to say thank you for this... I am not using podman, but instead wireguard that is supported directly in the later udm pro versions kernel.

I was facing a very similar issue.. would connect/handshake, but couldn't access anything. I spent hours trying to figure out what the heck was going on.. I still don't quite understand it, but removing the 'Address' line under the '[Interface]' section worked for me as well. Everything just started working...

Can't tell you how much time I wasted quadruple checking all my firewall rules in udm and other settings.

Thanks! @turowicz

turowicz commented 2 years ago

@up2geek I've been through the same. This is all trail and error discovery to make it work.

turowicz commented 2 years ago

@up2geek how to use wireguard without podman?

Slychocobo commented 1 year ago

Again, Also had this Exact same problem, you've stopped me pulling my hair out! Thank you @turowicz

turowicz commented 1 year ago

Nowadays I also remove the DNS entry on client.