Bug: Kubernetes services cannot be resolved due to DNS overrides and routing conflicts

[wolph]

TLDR: Kubernetes services cannot be resolved anymore because the DNS configuration is being overwritten

1. Is this urgent?

   • [ ] Yes
   • [x] No

2. What VPN service provider are you using?

   • [ ] PIA
   • [ ] Mullvad
   • [ ] Windscribe
   • [x] Surfshark
   • [ ] Cyberghost

3. What's the version of the program?

Running version latest built on 2020-07-09T11:57:17Z (commit dc1c7ea)

4. What are you using to run the container?

   • [ ] Docker run
   • [ ] Docker Compose
   • [x] Kubernetes
   • [ ] Docker stack
   • [ ] Docker swarm
   • [ ] Podman
   • [ ] Other:

5. Extra information

Logs:

DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: disabled
 |--Block malicious: disabled
 |--Block surveillance: disabled
 |--Block ads: disabled

...

2020-07-09T13:22:42.163Z        INFO    firewall configurator: accepting any input traffic on port 8888
2020-07-09T13:22:42.163Z        INFO    http server: listening on 0.0.0.0:8000
2020-07-09T13:22:42.163Z        INFO    dns configurator: using DNS address 1.1.1.1 internally
2020-07-09T13:22:42.163Z        INFO    dns configurator: using DNS address 1.1.1.1 system wide
2020-07-09T13:22:42.163Z        INFO    openvpn configurator: writing auth file /etc/openvpn/auth.conf
2020-07-09T13:22:42.164Z        INFO    openvpn configurator: starting openvpn
2020-07-09T13:22:42.166Z        INFO    openvpn: OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
2020-07-09T13:22:42.166Z        INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-07-09T13:22:42.167Z        INFO    tinyproxy configurator: starting tinyproxy server
2020-07-09T13:22:42.168Z        INFO    openvpn: WARNING: you are using user/group/chroot/setcon without persist-tun -- this may cause restarts to fail
2020-07-09T13:22:42.170Z        INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]81.19.209.124:1194
2020-07-09T13:22:42.170Z        INFO    openvpn: UDP link local: (not bound)
2020-07-09T13:22:42.170Z        INFO    openvpn: UDP link remote: [AF_INET]81.19.209.124:1194
2020-07-09T13:22:42.170Z        INFO    openvpn: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1633', remote='link-mtu 1581'
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'auth' is used inconsistently, local='auth SHA512', remote='auth [null-digest]'
2020-07-09T13:22:42.182Z        INFO    openvpn: [nl-ams-v024.prod.surfshark.com] Peer Connection Initiated with [AF_INET]81.19.209.124:1194

Configuration file:

apiVersion: v1                                
kind: Pod                                     
metadata:                                     
  name: vpn-test
  namespace: default                          
spec:                                         
  containers:                                 
  - name: shell
    image: ubuntu
    command: ['bash']                         
    stdin: true                               
    tty: true                                 
  - name: proxy                               
    env:                                      
      - name: USER                            
        value: ...
      - name: PASSWORD                        
        value: ...
      - name: VPNSP                           
        value: 'surfshark'                    
      - name: FIREWALL                        
        value: 'off'                           
      - name: EXTRA_SUBNETS                   
        value: '10.192.0.0/9'
      - name: SHADOWSOCKS                     
        value: 'on'                          
      - name: TINYPROXY                       
        value: 'on'                           
      - name: DOT                             
        value: 'on'                           
      - name: DOT_CACHING                     
        value: 'off'                          
      - name: BLOCK_MALICIOUS                 
        value: 'off'                          
      - name: DNS_UPDATE_PERIOD               
        value: '0'                            
    image: qmcgaw/private-internet-access     
    imagePullPolicy: Always                   
    ports:                                    
      - containerPort: 8888                   
      - containerPort: 8388                   
      - containerPort: 8388                   
        protocol: UDP                         
      - containerPort: 8000                   
    securityContext:                          
      privileged: true                        
      capabilities:                           
        add:                                  
          - NET_ADMIN                         

Host OS: DigitalOcean Kubernetes cluster

I believe that svc.cluster.local should be added to the search parameter in /etc/resolv.conf and that unbound needs to use the internal k8s dns server to resolve those local domainnames.

Running in a normal pod:

root@shell:/# nslookup kube-dns.kube-system
Server:         10.245.0.10
Address:        10.245.0.10#53

Name:   kube-dns.kube-system.svc.cluster.local
Address: 10.245.0.10

Running with the VPN sidecar:

root@shell:/# nslookup kube-dns.kube-system
;; Got SERVFAIL reply from 127.0.0.1, trying next server
;; Got SERVFAIL reply from 127.0.0.1, trying next server
Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find kube-dns.kube-system.default.svc.cluster.local: SERVFAIL

Running from the VPN sidecar:

/ # nslookup kube-dns.kube-system
Server:         127.0.0.1
Address:        127.0.0.1:53

** server can't find kube-dns.kube-system: SERVFAIL

** server can't find kube-dns.kube-system: SERVFAIL

[graytonio]

The cluster CIDR is 10.43.0.0/16 I opened it up to all for testing. The kube-dns server service was at 10.43.0.10 doing any lookups for cluster domains like podinfo.namespace.svc.cluster.local without the firewall outbound would fail but if I added it they started working again.

[qdm12]

@graytonio FIREWALL_OUTBOUND_SUBNETS does change routing, so it is a routing+firewall issue, not just a firewall issue. Isn't 10.0.0.0/8 the local network the gluetun pod is part of already? By default Gluetun configures routing and the firewall to allow communication with local subnets, it's strange it did not allow it for the cluster DNS server. Maybe exec in Gluetun ip route and check the difference with/without FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8?

On a related note, DNS_KEEP_NAMESERVER=on now does not change any DNS settings (on the latest image), although at your own risk since this will have DNS traffic go outside the VPN tunnel. Not a great solution, but a viable one for Kubernetes it seems.

[tom-elvidge]

The combination of FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8 and DNS_PLAINTEXT_ADDRESS=10.43.0.10 has dns resolution for k8s services working for me. Not a fan of hard coding the kube-dns ip here but it does the job.

[asininemonkey]

Going even simpler, using the advice above and some local testing, this also just works for me (using k3s):

env:
  - name: DNS_KEEP_NAMESERVER
    value: on
  - name: FIREWALL_OUTBOUND_SUBNETS
    value: 10.43.0.0/16 # https://docs.k3s.io/cli/server#networking

Looking at ip route in the gluetun container I see:

<<K9s-Shell>> Pod: debugger/debugger-7d4f4c6bf-42d5w | Container: gluetun
/ # ip route
default via 10.42.0.1 dev eth0
10.42.0.0/24 dev eth0 proto kernel scope link src 10.42.0.172
10.42.0.0/16 via 10.42.0.1 dev eth0