search

ngoduykhanh / wireguard-ui

Wireguard web interface
MIT License
4.08k stars 505 forks source link

Does this work with Kubernetes? #166

Closed Slyke closed 2 years ago

Slyke commented 2 years ago

Hello, having some issues getting this up and running on a K8s cluster. I currently have the plain linuxserver/wireguard image working, I've had to add the following commands to the post up and down events:

PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -A FORWARD -o %i -j ACCEPT; iptables -t nat -A POSTROUTING -o eno1 -j MASQUERADE; iptables -t nat -A POSTROUTING -o enp3s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -D FORWARD -o %i -j ACCEPT; iptables -t nat -D POSTROUTING -o eno1 -j MASQUERADE; iptables -t nat -D POSTROUTING -o enp3s0 -j MASQUERADE

When running ifconfig inside the linuxserver/wireguard container, I can see various network interfaces:

cni0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet XX.XX.XX.XX  netmask 255.255.255.0  broadcast XX.XX.XX.XX
        inet6 fe80::XXXX:XXXX:XXXX:XXXX  prefixlen 64  scopeid 0x20<link>
        ether 76:51:a7:1a:e9:fa  txqueuelen 1000  (Ethernet)
        RX packets 716109126  bytes 1423475819117 (1.4 TB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 931484216  bytes 1198413121574 (1.1 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:5b:be:7c:5f  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet XX.XX.XX.XX  netmask 255.255.0.0  broadcast XX.XX.XX.XX
        inet6 fe80::XXXX:XXXX:XXXX:XXXX  prefixlen 64  scopeid 0x20<link>
        ether 9c:8e:99:de:XX:XX  txqueuelen 1000  (Ethernet)
        RX packets 1483221502  bytes 1291132525561 (1.2 TB)
        RX errors 0  dropped 8186  overruns 0  frame 0
        TX packets 1563818673  bytes 1585395647345 (1.5 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 20  memory 0xfe500000-fe520000

flannel.1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet 10.XXX.4.0  netmask 255.255.255.255  broadcast 10.XXX.4.0
        inet6 fe80::d446:a0ff:fecb:7ea6  prefixlen 64  scopeid 0x20<link>
        ether d6:46:a0:cb:7e:a6  txqueuelen 0  (Ethernet)
        RX packets 481994413  bytes 256665655255 (256.6 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 400905273  bytes 1254230417997 (1.2 TB)
        TX errors 0  dropped 2545 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 160553  bytes 11459221 (11.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 160553  bytes 11459221 (11.4 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth15c5f724: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::a859:c2ff:fe17:3f26  prefixlen 64  scopeid 0x20<link>
        ether aa:59:c2:17:3f:26  txqueuelen 0  (Ethernet)
        RX packets 7142683  bytes 703168943 (703.1 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 7505770  bytes 750273775 (750.2 MB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth260ff743: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1450
        inet6 fe80::349a:2bff:feac:dd66  prefixlen 64  scopeid 0x20<link>
        ether 36:9a:2b:ac:dd:66  txqueuelen 0  (Ethernet)
        RX packets 6476739  bytes 2182348411 (2.1 GB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5085700  bytes 1627565134 (1.6 GB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

... and many more veths

But when I run ifconfig inside the wireguard-ui container, I just get:

eth0      Link encap:Ethernet  HWaddr 1E:86:43:XX:XX:XX
          inet addr:10.XXX.1.25  Bcast:10.244.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1450  Metric:1
          RX packets:5293 errors:0 dropped:0 overruns:0 frame:0
          TX packets:354 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:338981 (331.0 KiB)  TX bytes:2405696 (2.2 MiB)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Both the deployment configurations are similar and both have:

        securityContext:
          privileged: true
          capabilities:
            add:
              - NET_ADMIN
              - SYS_MODULE

My Post up and down scripts won't obviously work, and it appears it's not usable inside K8s.

ngoduykhanh commented 2 years ago

I haven't tried it on Kubernetes before but the ifconfig output of linuxserver/wireguard container looks weird to me. It looks like the output of the Kubernetes node instead. Anyway, those iptables rules should be applied on the machine where wireguard server is running, it is not always the wireguard-ui machine.

Slyke commented 2 years ago

Figured out the issue. I have my K8s cluster running on a mix of hardware. Some RPis, some desktops and some ITX machines. Depending on the OS, it could use eth0, eth1, en0 etc as the interface. I just decided which nodes I wanted it to run on (they all have the same interface name) and instructed K8s to only spawn it on those.

Thank you!