Wireguard non-functional

[KittyKatt]

Hey there, I've had edge3 installed for some time now on my UDM Pro. Over the last weeks, my UDM auto updated to 1.11.4 firmware (which I thought I disabled? too late now lol). The first thing I went about doing was figuring out why my multipath routing wasn't working anymore and come to find that the routes are being rejected, which led me to believe I was back on stock kernel. Sure enough. So, I rebooted into edge3 with the newest udm-kernel-tools package and I have multipath routing back! Yay!

However, since this reboot wireguard has been totally non-functional. I get this when attempting to bring up the wg0 interface:

Warning: `/mnt/data/wireguard/etc/wireguard/wg0.conf' is world accessible
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.10.10.2 dev wg0
[#] ip link set mtu 1420 up dev wg0
Error: mtu less than device minimum.
[#] ip link delete dev wg0

After tons of trying, I rebooted back into the stock kernel and tried to use the wireguard-kmod utility again. It loaded the kernel modules in there and functions great, but I then have no multipathing.

This worked right up until the auto update to v1.11.4. Is there any reason this could now be broken for me?

For reference, the stock kernel is: 4.19.152-al-linux-v10.2.0-v1.11.4.3940-e66d85f

[KittyKatt]

Additionally, I actually had to edit the 10-setup_wireguard on_boot configuration to include the following, because it wasn't finding any kernel modules for wireguard whatsoever before adding this:

   elif [ -e /lib/modules/$ver/kernel/net/wireguard/wireguard.ko ]; then
     modprobe wireguard

This script worked before the update to v1.11.4, so I'm unsure HOW it was functioning before this.

[fabianishere]

Did you check whether the module is actually loaded? (e.g., lsmod)

[KittyKatt]

lsmod shows that the kernel module is loaded.

Module                  Size  Used by
wireguard             126976  0
ip6_udp_tunnel         16384  1 wireguard
udp_tunnel             16384  1 wireguard
xt_conntrack           16384  8
nf_nat_tftp            16384  0
nf_conntrack_tftp      16384  1 nf_nat_tftp
nf_nat_pptp            16384  0
nf_conntrack_pptp      16384  1 nf_nat_pptp
nf_nat_h323            20480  0
nf_conntrack_h323      53248  1 nf_nat_h323
nf_nat_proto_gre       16384  1 nf_nat_pptp
nf_conntrack_proto_gre    16384  1 nf_conntrack_pptp
nf_nat_ftp             16384  0
nf_conntrack_ftp       16384  1 nf_nat_ftp
nf_log_ipv4            16384  2
nf_log_common          16384  1 nf_log_ipv4
xt_LOG                 16384  2
nf_app                 16384  0
t_miner                49152  1 nf_app
tdts                  438272  1 t_miner
xt_dpi                 16384  0
ifb                    16384  0
sch_htb                24576  0
ppp_generic            36864  0
slhc                   20480  1 ppp_generic
lzo                    16384  4
lzo_compress           16384  1 lzo
zram                   24576  1
gpiodev                16384  0
ubnthal              1253376  0
ubnt_common           102400  1 ubnthal

[fabianishere]

Could you check what the minimum MTU is for wg0? Perhaps that might give some hint.

[KittyKatt]

I did:

ip link add wg0 type wireguard
ip -4 address add 10.10.10.2 dev wg0

ip addr:

34: wg0: <PROMISC> mtu 2147483552 qdisc noop state DOWN group default qlen 1000
    link/infiniband 
    inet 10.10.10.2/32 scope global wg0
       valid_lft forever preferred_lft forever

That's a pretty wild MTU.

[KittyKatt]

Some additional information about the link:

36: wg0: <PROMISC> mtu 2147483552 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/infiniband  promiscuity 0 
    wireguard addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

[fabianishere]

Very interesting 🤔 What also surprises me is that it reports a link over Infiniband. Since it worked with the previous firmware, I suspect that there are some incompatibilities between Ubiquiti's proprietary kernel modules and the edge3 kernel.

I am working on developing shims for these module, so that this problem should not happen as often. I have sent Ubiquiti a request for the latest kernel sources so that I can update the custom kernel.

In the meantime, you could perhaps downgrade your firmware installation back to v1.11.0 if you need this to work.

[KittyKatt]

I will probably end up downgrading for now. Could you keep this issue open and tag me when you've got the latest source figured out and a new custom kernel built?

[fabianishere]

Hi @KittyKatt, I managed to reproduce your issue and verify it with the new kernel sources. It appears the kernel itself is not the problem (there were little changes for v1.11.4). Instead, what happens is that modprobe selects the incorrect wireguard.ko file (which is now included in the stock firmware).

You can verify this yourself as follows:

insmod /lib/modules/4.19.152-edge3/kernel/net/wireguard/wireguard.ko

This should load the right kernel module. When creating a new interface now, Linux should report the expected values.

I'll fix the issue in udm-kernel-tools so that this will not happen anymore.

[fabianishere]

Fixed with https://github.com/fabianishere/udm-kernel-tools/pull/65

[KittyKatt]

@fabianishere Just saw this. I'm going to hopefully be trying this out and validating this weekend. Thank you!