Wireguard chain connection issue

[deg749Goroe3]

Hello,

I have faced a connection issue when using the Wireguard chain (2 servers) with udp2raw. Udp2raw receives packets until the Wireguard server (same server as udp2raw) is connected to Wireguard server 2. After that, I saw that the server received a syn from the client but did not send it back to the client.

This rule helps, but the client has a dynamic IP, which is not a solution.

ip rule add to client_ip/24 table main

Can anybody guide me on how to route all udp2raw traffic to the default network gateway (eth0 in my case)?

[deg749Goroe3]

root@:~# ip route
default via masked_server_ip dev eth0 proto static
10.15.10.0/24 dev wgchainclient proto kernel scope link src 10.15.10.2
10.28.188.0/24 dev wg0 proto kernel scope link src 10.28.188.1
masked_server_ip/24 dev eth0 proto kernel scope link src masked_server_ip

root@:~# ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet masked_ip  netmask 255.255.255.0  broadcast masked_ip
        inet6 masked_ip  prefixlen 64  scopeid 0x20<link>
        ether masked_mac  txqueuelen 1000  (Ethernet)
        RX packets 14365  bytes 1069154 (1.0 MB)
        RX errors 0  dropped 3989  overruns 0  frame 0
        TX packets 8977  bytes 1519923 (1.5 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 172  bytes 13584 (13.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 172  bytes 13584 (13.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wg0: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1280
        inet 10.28.188.1  netmask 255.255.255.0  destination 10.28.188.1
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 2  bytes 296 (296.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2  bytes 184 (184.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

wgchainclient: flags=209<UP,POINTOPOINT,RUNNING,NOARP>  mtu 1280
        inet 10.15.10.2  netmask 255.255.255.0  destination 10.15.10.2
        unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  txqueuelen 1000  (UNSPEC)
        RX packets 724  bytes 55116 (55.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7024  bytes 1059012 (1.0 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@:~# iptables-save
# Generated by iptables-save v1.8.7 on Mon Sep 18 16:16:39 2023
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:udp2rawDwrW_77e42f96_C0 - [0:0]
-A INPUT -p tcp -m tcp --dport 8888 -j udp2rawDwrW_77e42f96_C0
-A FORWARD -i wg0 -j ACCEPT
-A FORWARD -o wg0 -j ACCEPT
-A udp2rawDwrW_77e42f96_C0 -j DROP
COMMIT
# Completed on Mon Sep 18 16:16:39 2023
# Generated by iptables-save v1.8.7 on Mon Sep 18 16:16:39 2023
*nat
:PREROUTING ACCEPT [7925:468407]
:INPUT ACCEPT [1303:60890]
:OUTPUT ACCEPT [58:3854]
:POSTROUTING ACCEPT [43:2805]
-A POSTROUTING -s 10.28.188.0/24 -o eth0 -m comment --comment wireguard-nat-rule -j MASQUERADE
-A POSTROUTING -o wgchainclient -j MASQUERADE
COMMIT
# Completed on Mon Sep 18 16:16:39 2023

root@:~# netstat -nlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      618/udp2raw_amd64
tcp        0      0 127.0.0.1:6010          0.0.0.0:*               LISTEN      1165/sshd: root@pts
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      561/systemd-resolve
tcp        0      0 0.0.0.0:1022            0.0.0.0:*               LISTEN      670/sshd: /usr/sbin
tcp6       0      0 ::1:6010                :::*                    LISTEN      1165/sshd: root@pts
tcp6       0      0 :::1022                 :::*                    LISTEN      670/sshd: /usr/sbin
udp        0      0 0.0.0.0:47500           0.0.0.0:*                           -
udp        0      0 127.0.0.53:53           0.0.0.0:*                           561/systemd-resolve
udp        0      0 0.0.0.0:51820           0.0.0.0:*                           -
udp6       0      0 :::47500                :::*                                -
udp6       0      0 :::51820                :::*                                -
raw        0      0 0.0.0.0:255             0.0.0.0:*               7           618/udp2raw_amd64
raw6       0      0 :::58                   :::*                    7           559/systemd-network
Active UNIX domain sockets (only servers)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     15226    1/init               /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     STREAM     LISTENING     15229    1/init               /run/systemd/fsck.progress
unix  2      [ ACC ]     STREAM     LISTENING     15240    1/init               /run/systemd/journal/stdout
unix  2      [ ACC ]     SEQPACKET  LISTENING     15243    1/init               /run/udev/control
unix  2      [ ACC ]     STREAM     LISTENING     20173    1169/systemd         /run/user/0/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     20180    1169/systemd         /run/user/0/bus
unix  2      [ ACC ]     STREAM     LISTENING     15228    1/init               @/org/kernel/linux/storage/multipathd
unix  2      [ ACC ]     STREAM     LISTENING     20182    1169/systemd         /run/user/0/gnupg/S.dirmngr
unix  2      [ ACC ]     STREAM     LISTENING     20184    1169/systemd         /run/user/0/gnupg/S.gpg-agent.browser
unix  2      [ ACC ]     STREAM     LISTENING     20186    1169/systemd         /run/user/0/gnupg/S.gpg-agent.extra
unix  2      [ ACC ]     STREAM     LISTENING     20188    1169/systemd         /run/user/0/gnupg/S.gpg-agent.ssh
unix  2      [ ACC ]     STREAM     LISTENING     20190    1169/systemd         /run/user/0/gnupg/S.gpg-agent
unix  2      [ ACC ]     STREAM     LISTENING     20192    1169/systemd         /run/user/0/pk-debconf-socket
unix  2      [ ACC ]     STREAM     LISTENING     20194    1169/systemd         /run/user/0/snapd-session-agent.socket
unix  2      [ ACC ]     STREAM     LISTENING     15327    352/systemd-journal  /run/systemd/journal/io.systemd.journal
unix  2      [ ACC ]     STREAM     LISTENING     17307    1/init               /var/snap/lxd/common/lxd-user/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     17305    1/init               /var/snap/lxd/common/lxd/unix.socket
unix  2      [ ACC ]     STREAM     LISTENING     17007    561/systemd-resolve  /run/systemd/resolve/io.systemd.Resolve
unix  2      [ ACC ]     STREAM     LISTENING     17287    1/init               /run/dbus/system_bus_socket
unix  2      [ ACC ]     STREAM     LISTENING     17309    1/init               /run/snapd.socket
unix  2      [ ACC ]     STREAM     LISTENING     17311    1/init               /run/snapd-snap.socket
unix  2      [ ACC ]     STREAM     LISTENING     17313    1/init               /run/uuidd/request
unix  2      [ ACC ]     STREAM     LISTENING     17304    1/init               @ISCSIADM_ABSTRACT_NAMESPACE
unix  2      [ ACC ]     STREAM     LISTENING     15215    1/init               /run/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     15217    1/init               /run/systemd/userdb/io.systemd.DynamicUser
unix  2      [ ACC ]     STREAM     LISTENING     15218    1/init               /run/systemd/io.system.ManagedOOM

0:      from all lookup local
32763:  from masked_server_ip/24 lookup main
32764:  from all lookup main suppress_prefixlength 0
32765:  not from all fwmark 0xca6c lookup 51820
32766:  from all lookup main
32767:  from all lookup default

[wangyu-]

Wireguard chain (2 servers)

what is a wireguard chain (2 servers)? can you make a diagram of your topology?

below is a tutorial of udp2raw+openvpn, which should be helpful for wireguard as well:

https://github.com/wangyu-/udp2raw/wiki/udp2raw-openvpn-config-guide

in the end there is a section about transparently redirect traffic

[deg749Goroe3]

Wireguard chain (2 servers)

what is a wireguard chain (2 servers)? can you make a diagram of your topology?

below is a tutorial of udp2raw+openvpn, which should be helpful for wireguard as well:

https://github.com/wangyu-/udp2raw/wiki/udp2raw-openvpn-config-guide

in the end there is a section about transparently redirect traffic

Client -> wireguard (client) -> udp2raw (client) -> udp2raw (server) -> wireguard server 1 -> Wireguard server 2

All works fine until Wireguard server 1 connects to Wireguard server 2 with wg-quick@wgclient2. After that, all udp2raw traffic goes through the wgclient2 interface. I have read the guide you sent, but I can't figure it out.

[wangyu-]

Client -> wireguard (client) -> udp2raw (client) -> udp2raw (server) -> wireguard server 1 -> Wireguard server 2

does this setting work well before you put udp2raw into the chain?

if so, could you please explain a bit how udp2raw is making it harder to work?

wireguard server 1 -> Wireguard server 2

how is it going here? I guess the server 1 is running two wireguards, 1 as client and 1 as server. So you have two wg interface on server 1. Is this correct?

root@:~# ip route default via masked_server_ip dev eth0 proto static 10.15.10.0/24 dev wgchainclient proto kernel scope link src 10.15.10.2 10.28.188.0/24 dev wg0 proto kernel scope link src 10.28.188.1 masked_server_ip/24 dev eth0 proto kernel scope link src masked_server_ip

root@:~# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet masked_ip netmask 255.255.255.0 broadcast masked_ip inet6 masked_ip prefixlen 64 scopeid 0x20

you posted some info, but you didn't mention on which machine you are running them. There is too much guess work for me. Can you add the missing info?

Also it will be helpful if you post all your udp2raw commands and wireguard confs.

[deg749Goroe3]

does this setting work well before you put udp2raw into the chain? Yes, Wireguard works fine without udp2raw in chain, and udp2raw works fine when Wireguard client don't enable on Server 1.

how is it going here? I guess the server 1 is running two wireguards, 1 as client and 1 as server. So you have two wg interface on server 1. Is this correct? Correct.

Server using Ubuntu 22.04.3 LTS (GNU/Linux 5.15.0-83-generic x86_64)

udp2raw server config (Located on Server 1):

-s
-l 0.0.0.0:8888
-r 127.0.0.1:51820
-k secret key
--auth-mode simple
--raw-mode faketcp
-a
--fix-gro
--cipher-mode xor

Wg server config (Located on Server 1):

[Interface]
PrivateKey = [key]
Address = 10.28.188.1/24
MTU = 1280
ListenPort = 51820

PreUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o wgchainclient -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o wgchainclient -j MASQUERADE

[Peer]
PublicKey = [key]
AllowedIPs = 10.28.188.2/32

Wg client config (Located on Server 1):

[Interface]
PrivateKey = [key]
Address = 10.15.10.2/24
DNS = 1.1.1.1, 1.0.0.1
MTU = 1280

PostUp = ip rule add from  server1_ip/24 table main    # this rule adding for bypass SSH access after client connects
PostDown = ip rule del from  server1_ip/24 table main

[Peer]
PublicKey = [key]
AllowedIPs = 0.0.0.0/0
Endpoint = wg_server2_ip:51821
PersistentKeepalive = 15

[deg749Goroe3]

I've tried to route udp2raw.service via custom namespace, which connected to eth0 (default gateway), but it's too complicated for me now =)

[deg749Goroe3]

Can I fwmark udp2raw traffic via iptables and add route to the main table?

ip rule add fwmark 0x64 table main

[wangyu-]

(I am not really very familiar with wireguard config)

In your Wg client config (Located on Server 1) it has:

AllowedIPs = 0.0.0.0/0

I guess this means: once the wg client on server1 establish connection with the server2, the default route on server1 will be changed, and all traffic will go through server2 by default. Then this route change will hijack udp2raw_server's traffic, so it's causing problem.

I guess the best practice is not to change the default route on server1. You instead add some specific rule to redict traffic from wg0 to wgchainclient. In this way, you won't have udp2raw or ssh's traffic being hijacked and causing weird problems.

Can I fwmark udp2raw traffic via iptables and add route to the main table? ip rule add fwmark 0x64 table main

This might not work. Since udp2raw is sending/receving packet at a low level, it's known to not work well together with iptables's fwmark. Methods based on marking udp2raw's traffic usually doesn't work.

I guess the best practice is not to change the default route on server1. You add some rule to redict traffic from wg0 to wgchainclient.

If you insist on change the default route on server1. check the --lower-level option. If set correctly, it will bypass any iptables and ip route rules, sending packet directly to the network interface.

[wangyu-]

Client -> wireguard (client) -> udp2raw (client) -> udp2raw (server) -> wireguard server 1 -> Wireguard server 2

Your goal is to let client's traffic go through wireguard sever2. Changing the default route of server1 is not necessary.

I personally think changing the default route of a remote server is a bad practice and a source of trouble. Avoid whenever possible.

[deg749Goroe3]

Client -> wireguard (client) -> udp2raw (client) -> udp2raw (server) -> wireguard server 1 -> Wireguard server 2

Your goal is to let client's traffic go through wireguard sever2. Changing the default route of server1 is not necessary.

I personally think changing the default route of a remote server is a bad practice and a source of trouble. Avoid whenever possible.

Yeah, you're right.

I've tried to change Allowed IPs to

AllowedIPs = 10.15.10.0/24

but has the server 1 IP after check.

Can you guide me on what I'm doing wrong?

[deg749Goroe3]

Also, the --lower-level option works just fine. Thank you very much.