[pod-gateway] Cannot reach external servers when using wireguard

ameyp commented 2 years ago

Helm chart name

pod-gateway

Helm chart version

latest

Container name

pod-gateway

Container tag

v1.6.0

Description

When I enable wireguard, I see in the corresponding container logs that wireguard seems to be up. However, I'm unable to send any traffic to hosts external to my network: nslookup and wget both timeout and fail.

I'm not sure how wireguard works, so please let me know what additional debug information I can provide. Here's ip addr:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 7a:23:27:ae:1f:f9 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.42.4.70/24 brd 10.42.4.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::7823:27ff:feae:1ff9/64 scope link 
       valid_lft forever preferred_lft forever
3: vxlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 1e:c0:77:5e:84:70 brd ff:ff:ff:ff:ff:ff
    inet 172.16.0.1/24 scope global vxlan0
       valid_lft forever preferred_lft forever
    inet6 fe80::1cc0:77ff:fe5e:8470/64 scope link 
       valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1370 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.6.0.2/32 scope global wg0
       valid_lft forever preferred_lft forever

and ip route:

default via 10.42.4.1 dev eth0 
10.0.0.0/8 via 10.42.4.1 dev eth0 
10.42.0.0/16 via 10.42.4.1 dev eth0 
10.42.4.0/24 dev eth0 proto kernel scope link src 10.42.4.70 
172.16.0.0/24 dev vxlan0 proto kernel scope link src 172.16.0.1 
192.168.0.0/16 via 10.42.4.1 dev eth0

Expected result

I'm able to send traffic to hosts outside of my network

Helm values to reproduce

image:
  tag: v1.6.0

routed_namespaces:
  - sabnzbd
  - sonarr

addons:
  vpn:
    enabled: true
    type: wireguard
    #configFileSecret: openvpn
    configFile: |-
      [Interface]
      PrivateKey = ActualPrivateKeyHere
      Address = 10.6.0.2/32
      DNS = 103.86.96.100, 103.86.99.100

      [Peer]
      PublicKey = ActualPublicKeyHere
      AllowedIPs = 0.0.0.0/0
      Endpoint = us1000.nordvpn.com:51820
    networkPolicy:
      enabled: true

      egress:
        - to:
          - ipBlock:
              cidr: 0.0.0.0/0
          ports:
            # VPN traffic port - change if your provider uses a different port
          - port: 51820
            protocol: UDP
        - to:
          - ipBlock:
              cidr: 192.168.0.0/16
        - to:
            # Allow traffic within K8S - change if your K8S cluster uses a different CIDR
          - ipBlock:
              cidr: 10.0.0.0/8

settings:
  # tun0 for openvpn, wg0 for wireguard
  VPN_INTERFACE: wg0
  # Prevent non VPN traffic to leave the gateway
  VPN_BLOCK_OTHER_TRAFFIC: true
  # If VPN_BLOCK_OTHER_TRAFFIC is true, allow VPN traffic over this port
  VPN_TRAFFIC_PORT: 51820
  # Traffic to these IPs will be send through the K8S gateway
  # change if your K8S cluster or home network uses a different CIDR
  VPN_LOCAL_CIDRS: "10.0.0.0/8 192.168.0.0/16"
  NOT_ROUTED_TO_GATEWAY_CIDRS: "10.0.0.0/8 192.168.0.0/16"

Additional Information

No response

Repo link

No response

ghost commented 2 years ago

:wave: @ameyp, we use the issue tracker exclusively for bug reports and feature requests. However, this issue appears to be a support request. Please use our support channels to get help.

ameyp commented 2 years ago

@bjw-s I got this to work by manually replacing the k8s-at-home wireguard container with https://docs.linuxserver.io/images/docker-wireguard. It wasn't a straightforward replacement since your helm chart tries to mount the config file at /etc/wireguard whereas theirs expects it to be at /config/.

FWIW I honestly think this is a bug in your container/helm chart and not a configuration issue at my end.

k8s-at-home / charts