search

docker / for-mac

Bug reports for Docker Desktop for Mac
https://www.docker.com/products/docker#/mac
2.44k stars 118 forks source link

Wireguard on Docker for Mac version 3.3.1 is not working #5627

Closed Dionid closed 3 years ago

Dionid commented 3 years ago

Expected behavior

I wanted to use wireguard for docker as I did in Docker for Mac version 3.2.2 where it works perfectly.

Actual behavior

But in latest version (after Linux Kernel update) it fails on first step:

$ ip link add dev wg0 type wireguard
Error: Unknown device type.

Information

Steps to reproduce the behavior

  1. Install v 3.3.1

  2. Create somewhere file wg0.conf with:

    ```
[Interface]
Address = 10.13.13.1
ListenPort = 51820
PrivateKey = AByrV+HTOsYWNVxdCmxIH02AEzY4EuL5+Rw/e6yK0W0=
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eth0 -j MASQUERADE

[Peer]
# peer1
PublicKey = e52FYYIXvFiT/rXuF/Wr/4elvU9YjWgfR3pegcz+NDc=
AllowedIPs = 10.13.13.2/32

[Peer]
# peer2
PublicKey = QpMukzKSZF7e2Wg5Ujw7iKrtEmYKvrDiIdWFvtmOUEA=
AllowedIPs = 10.13.13.3/32
```
  3. After that run: docker run --cap-add net_admin --cap-add sys_module -v path/to/wg0.cong:/etc/wireguard -p 5555:5555/udp cmulk/wireguard-docker:buster

In Docker for mac v3.2.2 it will setup with no problems, in Docker for mac version 3.3.1 it will crush on first command like so:

$ ip link add dev wg0 type wireguard
Error: Unknown device type.

Additional question

If I'm not mistaken, on linux kernel higher 5.6 wireguard must be installed in kernel itself, is it true for Docker for mac version 3.3.1?

enovella commented 3 years ago

Hi all,

I do confirm that I've been experimenting the same behavior on MacBook Pro Intel chips with Docker Desktop 3.3.1. The Docker image was using a Ubuntu20.04 with a Linux kernel v5.10.

Previous versions of Docker Desktop (3.2.x) were working flawlessly with the WireGuard VPN server running inside of it. What I've read on the internet, it is related to missing kernel headers to recompile the kernel module. However, Linux kernels >= 5.6 should contain the WireGuard code included.

Best

enovella commented 3 years ago

It seems that the version 3.3.0 added these upgrades:

enovella commented 3 years ago

When building the image, the kernel module is built for the kernel 5.4.0-72-generic:

 => [2/7] RUN apt update -y && apt upgrade -y                                                                                                                                          3.3s
 => [3/7] RUN apt install iptables iproute2 wireguard -y                                                                                                                               31.0s
 => => # Selecting previously unselected package linux-headers-5.4.0-72.
 => => # Preparing to unpack .../56-linux-headers-5.4.0-72_5.4.0-72.80_all.deb ...
 => => # Unpacking linux-headers-5.4.0-72 (5.4.0-72.80) ...
 => => # Selecting previously unselected package linux-headers-5.4.0-72-generic.
 => => # Preparing to unpack .../57-linux-headers-5.4.0-72-generic_5.4.0-72.80_amd64.deb ...
 => => # Unpacking linux-headers-5.4.0-72-generic (5.4.0-72.80) ...
...
 => => # Loading new wireguard-1.0.20201112 DKMS files...
 => => # It is likely that 5.10.25-linuxkit belongs to a chroot's host
 => => # Building for 5.4.0-72-generic

Running the instance of the dockerized WireGuard VPN server:

docker run --cap-add net_admin --cap-add sys_module -p 1194:1194/udp wg-docker
[#] ip link add wg0 type wireguard
Error: Unknown device type.
Unable to access interface: Protocol not supported
[#] ip link delete dev wg0
Cannot find device "wg0"

Inside of the running container:

root@c60de13e3005:/# uname -a
Linux c60de13e3005 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

root@c60de13e3005:/# find /lib/modules/|grep wireguard
/lib/modules/5.4.0-72-generic/updates/dkms/wireguard.ko

root@c60de13e3005:/#  zgrep WIREGUARD /proc/config.gz
# CONFIG_WIREGUARD is not set

root@c60de13e3005:/# modinfo wireguard
modinfo: ERROR: Module alias wireguard not found.

root@c60de13e3005:/# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:    20.04
Codename:   focal

root@c60de13e3005:/# wg show
enovella commented 3 years ago

Checking the Linuxkit kernel config on 5.10.25:

git clone https://github.com/linuxkit/linuxkit
cd linuxkit/kernel

/tmp/linuxkit/kernel $ ls -lah|grep x86
-rw-r--r--    1 edu  wheel   109K Apr 30 09:35 config-4.19.x-x86_64
-rw-r--r--    1 edu  wheel   120K Apr 30 09:35 config-5.10.x-x86_64

The kernel config has WireGuard as module (m):

CONFIG_WIREGUARD=m
# CONFIG_WIREGUARD_DEBUG is not set
enovella commented 3 years ago

Reading this code proves that Linuxkit is checking if the kernel module needs to be built or not.

# WireGuard (skip kernels which have it in tree)
RUN if [  ! -d /linux/drivers/net/wireguard ]; then \
        curl -fsSL -o /wireguard.tar.xz "${WIREGUARD_URL}" && \
        echo "${WIREGUARD_SHA256}  /wireguard.tar.xz" | sha256sum -c - && \
        cp /wireguard.tar.xz /out/src/ && \
        tar -C / --one-top-level=wireguard --strip-components=2 -xJf /wireguard.tar.xz "wireguard-linux-compat-${WIREGUARD_VERSION}/src" && \
        make -j "$(getconf _NPROCESSORS_ONLN)" M="/wireguard" modules; \
     fi
enovella commented 3 years ago

The root cause of this is that WireGuard goes as kernel module on 5.10.x and Linuxkit is not providing kernel headers to compile wg as a module. Or the kernel flag CONFIG_WIREGUARD is not set:

root@18d88001de83:/# uname -a
Linux 18d88001de83 5.10.25-linuxkit #1 SMP Tue Mar 23 09:27:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@18d88001de83:/# find /lib/modules/|grep wireguard
/lib/modules/5.4.0-72-generic/updates/dkms/wireguard.ko
root@18d88001de83:/# apt install linux-headers-$(uname -r)
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package linux-headers-5.10.25-linuxkit
E: Couldn't find any package by glob 'linux-headers-5.10.25-linuxkit'
root@18d88001de83:/# modprobe wireguard
modprobe: FATAL: Module wireguard not found in directory /lib/modules/5.10.25-linuxkit
enovella commented 3 years ago

Recompiling the kernel v5.10.xx made me to ensure that WireGuard is included in the kernel as a module. Therefore, we need the kernel headers to build it.

[11:27 edu@xps kernel]  (master)>  docker run --rm -ti -v $(pwd):/src linuxkit/kconfig
/ # cd li
lib/             linux-4.19.188/  linux-5.10.31/   linux-5.11.15/   linux-5.4.113/
/ # cd linux-5.10.31/
/linux-5.10.31 # make menuconfig

Screenshot from 2021-05-03 11-26-39

RGafiyatullin commented 3 years ago

Some might find the following recipe handy:

ARG ALPINE_VERSION=3.13
ARG KERNEL_IMAGE=docker/for-desktop-kernel
ARG KERNEL_VERSION=5.10.25-6594e668feec68f102a58011bb42bd5dc07a7a9b

FROM alpine:${ALPINE_VERSION} AS alpine
FROM ${KERNEL_IMAGE}:${KERNEL_VERSION} AS kernel

FROM alpine:${ALPINE_VERSION} AS kmods

COPY --from=kernel / /kernel

RUN apk add --no-cache --update \
    abuild \
    autoconf \
    bc \
    bison \
    binutils \
    build-base \
    ca-certificates \
    cmake \
    coreutils \
    diffutils \
    findutils \
    flex \
    gcc  \
    git \
    grep \
    elfutils-dev \
    ncurses-dev \
    pciutils \
    usbutils \
    util-linux \
    wget \
    xz

RUN \
    export ARCH=$(uname -m) && \
    export KERNEL_RELEASE=$(uname -r) && \
    export KERNEL_SRC_D="/usr/src/linux-source-${KERNEL_RELEASE}/" && \
    cd / && \
    mkdir -p /usr/src && \
    xz -d < /kernel/linux.tar.xz | (cd /usr/src/; tar x) && \
    mv /usr/src/linux "$KERNEL_SRC_D" && \
    tar xf /kernel/kernel-headers.tar && \
    zcat /proc/1/root/proc/config.gz > "${KERNEL_SRC_D}/.config" 

RUN \
    export ARCH=$(uname -m) && \
    export KERNEL_RELEASE=$(uname -r) && \
    export KERNEL_SRC_D="/usr/src/linux-source-${KERNEL_RELEASE}/" && \
    cd $KERNEL_SRC_D && \
    /bin/sh scripts/config --module WIREGUARD && \
    /bin/sh scripts/config --module CRYPTO_CURVE25519_X86 && \
    /bin/sh scripts/config --module CRYPTO_BLAKE2S_X86 && \
    /bin/sh scripts/config --module CRYPTO_ARCH_HAVE_LIB_BLAKE2S && \
    /bin/sh scripts/config --module CRYPTO_LIB_BLAKE2S_GENERIC && \
    /bin/sh scripts/config --module CRYPTO_LIB_BLAKE2S && \
    /bin/sh scripts/config --module CRYPTO_LIB_CHACHA && \
    /bin/sh scripts/config --module CRYPTO_ARCH_HAVE_LIB_CURVE25519 && \
    /bin/sh scripts/config --module CRYPTO_LIB_CURVE25519_GENERIC && \
    /bin/sh scripts/config --module CRYPTO_LIB_CURVE25519 && \
    /bin/sh scripts/config --module CRYPTO_LIB_POLY1305 && \
    /bin/sh scripts/config --module CRYPTO_LIB_CHACHA20POLY1305

ARG MAKE_JOBS=16

RUN \
    export ARCH=$(uname -m) && \
    export KERNEL_RELEASE=$(uname -r) && \
    # export KERNEL_HEADERS_D="/usr/src/linux-headers-${KERNEL_RELEASE}/" && \
    export KERNEL_SRC_D="/usr/src/linux-source-${KERNEL_RELEASE}/" && \
    cd "${KERNEL_SRC_D}" && \
    make -j "${MAKE_JOBS}" modules

FROM alpine:latest

RUN mkdir -p /opt/wireguard-kmods

COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/lib/crypto/libchacha20poly1305.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/lib/crypto/libcurve25519-generic.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/arch/x86/crypto/curve25519-x86_64.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/lib/crypto/libblake2s-generic.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/arch/x86/crypto/blake2s-x86_64.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/lib/crypto/libblake2s.ko /opt/wireguard-kmods/
COPY --from=kmods /usr/src/linux-source-5.10.25-linuxkit/drivers/net/wireguard/wireguard.ko /opt/wireguard-kmods/

COPY entrypoint.sh /opt/wireguard-kmods/entrypoint.sh

WORKDIR /opt/wireguard-kmods
ENTRYPOINT /opt/wireguard-kmods/entrypoint.sh

where the entrypoint.sh is as follows:

#!/bin/sh

MODULES="libchacha20poly1305.ko libcurve25519-generic.ko curve25519-x86_64.ko libblake2s-generic.ko blake2s-x86_64.ko libblake2s.ko wireguard.ko"

if [ "$LOAD_MODULES" = "1" ]; then
    echo "Loading modules..."
    for module in $MODULES; do 
        echo "- ${module}"
        insmod $module
    done
    echo "Done"
else
    echo "Not loading modules"
fi

sleep infinity

The build-args corresponding to your docker-for-mac version can be found out this way:

KERNEL_IMAGE=$(docker run --rm --privileged --pid=host alpine:latest nsenter -t 1 -m -u -n -i cat /etc/linuxkit.yml | yq eval '.kernel.image' - | cut -d ':' -f 1)
KERNEL_VERSION=$(docker run --rm --privileged --pid=host alpine:latest nsenter -t 1 -m -u -n -i cat /etc/linuxkit.yml | yq eval '.kernel.image' - | cut -d ':' -f 2)
enovella commented 3 years ago

Working on Docker-for-mac v3.4.0 via loading kernel modules (compiled from above instructions):

Loading modules...
- libchacha20poly1305.ko
- libcurve25519-generic.ko
- curve25519-x86_64.ko
- libblake2s-generic.ko
- blake2s-x86_64.ko
- libblake2s.ko
- wireguard.ko
Done
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 10.0.0.1/29 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Luckily, maintainers integrate that on next releases. @stephen-turner

docker-robott commented 3 years ago

Issues go stale after 90 days of inactivity. Mark the issue as fresh with /remove-lifecycle stale comment. Stale issues will be closed after an additional 30 days of inactivity.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle stale

docker-robott commented 3 years ago

Closed issues are locked after 30 days of inactivity. This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows. /lifecycle locked