Bug: Kubernetes services cannot be resolved due to DNS overrides and routing conflicts

qdm12 / gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.

https://hub.docker.com/r/qmcgaw/gluetun

MIT License

7.23k stars 345 forks source link

Bug: Kubernetes services cannot be resolved due to DNS overrides and routing conflicts #188

Open wolph opened 4 years ago

wolph commented 4 years ago

TLDR: Kubernetes services cannot be resolved anymore because the DNS configuration is being overwritten

Is this urgent?
- [ ] Yes
- [x] No
What VPN service provider are you using?
- [ ] PIA
- [ ] Mullvad
- [ ] Windscribe
- [x] Surfshark
- [ ] Cyberghost
What's the version of the program?

Running version latest built on 2020-07-09T11:57:17Z (commit dc1c7ea)

What are you using to run the container?
- [ ] Docker run
- [ ] Docker Compose
- [x] Kubernetes
- [ ] Docker stack
- [ ] Docker swarm
- [ ] Podman
- [ ] Other:
Extra information

Logs:

DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: disabled
 |--Block malicious: disabled
 |--Block surveillance: disabled
 |--Block ads: disabled

...

2020-07-09T13:22:42.163Z        INFO    firewall configurator: accepting any input traffic on port 8888
2020-07-09T13:22:42.163Z        INFO    http server: listening on 0.0.0.0:8000
2020-07-09T13:22:42.163Z        INFO    dns configurator: using DNS address 1.1.1.1 internally
2020-07-09T13:22:42.163Z        INFO    dns configurator: using DNS address 1.1.1.1 system wide
2020-07-09T13:22:42.163Z        INFO    openvpn configurator: writing auth file /etc/openvpn/auth.conf
2020-07-09T13:22:42.164Z        INFO    openvpn configurator: starting openvpn
2020-07-09T13:22:42.166Z        INFO    openvpn: OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
2020-07-09T13:22:42.166Z        INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-07-09T13:22:42.167Z        INFO    tinyproxy configurator: starting tinyproxy server
2020-07-09T13:22:42.168Z        INFO    openvpn: WARNING: you are using user/group/chroot/setcon without persist-tun -- this may cause restarts to fail
2020-07-09T13:22:42.170Z        INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]81.19.209.124:1194
2020-07-09T13:22:42.170Z        INFO    openvpn: UDP link local: (not bound)
2020-07-09T13:22:42.170Z        INFO    openvpn: UDP link remote: [AF_INET]81.19.209.124:1194
2020-07-09T13:22:42.170Z        INFO    openvpn: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1633', remote='link-mtu 1581'
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
2020-07-09T13:22:42.182Z        INFO    openvpn: WARNING: 'auth' is used inconsistently, local='auth SHA512', remote='auth [null-digest]'
2020-07-09T13:22:42.182Z        INFO    openvpn: [nl-ams-v024.prod.surfshark.com] Peer Connection Initiated with [AF_INET]81.19.209.124:1194

Configuration file:

apiVersion: v1                                
kind: Pod                                     
metadata:                                     
  name: vpn-test
  namespace: default                          
spec:                                         
  containers:                                 
  - name: shell
    image: ubuntu
    command: ['bash']                         
    stdin: true                               
    tty: true                                 
  - name: proxy                               
    env:                                      
      - name: USER                            
        value: ...
      - name: PASSWORD                        
        value: ...
      - name: VPNSP                           
        value: 'surfshark'                    
      - name: FIREWALL                        
        value: 'off'                           
      - name: EXTRA_SUBNETS                   
        value: '10.192.0.0/9'
      - name: SHADOWSOCKS                     
        value: 'on'                          
      - name: TINYPROXY                       
        value: 'on'                           
      - name: DOT                             
        value: 'on'                           
      - name: DOT_CACHING                     
        value: 'off'                          
      - name: BLOCK_MALICIOUS                 
        value: 'off'                          
      - name: DNS_UPDATE_PERIOD               
        value: '0'                            
    image: qmcgaw/private-internet-access     
    imagePullPolicy: Always                   
    ports:                                    
      - containerPort: 8888                   
      - containerPort: 8388                   
      - containerPort: 8388                   
        protocol: UDP                         
      - containerPort: 8000                   
    securityContext:                          
      privileged: true                        
      capabilities:                           
        add:                                  
          - NET_ADMIN

Host OS: DigitalOcean Kubernetes cluster

I believe that svc.cluster.local should be added to the search parameter in /etc/resolv.conf and that unbound needs to use the internal k8s dns server to resolve those local domainnames.

Running in a normal pod:

root@shell:/# nslookup kube-dns.kube-system
Server:         10.245.0.10
Address:        10.245.0.10#53

Name:   kube-dns.kube-system.svc.cluster.local
Address: 10.245.0.10

Running with the VPN sidecar:

root@shell:/# nslookup kube-dns.kube-system
;; Got SERVFAIL reply from 127.0.0.1, trying next server
;; Got SERVFAIL reply from 127.0.0.1, trying next server
Server:         127.0.0.1
Address:        127.0.0.1#53

** server can't find kube-dns.kube-system.default.svc.cluster.local: SERVFAIL

Running from the VPN sidecar:

/ # nslookup kube-dns.kube-system
Server:         127.0.0.1
Address:        127.0.0.1:53

** server can't find kube-dns.kube-system: SERVFAIL

** server can't find kube-dns.kube-system: SERVFAIL

wolph commented 4 years ago

PS: ping is broken in alpine it seems. nslookup works but ping says "bad address" It might be related to ndots: https://github.com/docker/for-linux/issues/755 I've seen the reverse as well: https://unix.stackexchange.com/questions/124283/busybox-ping-ip-works-but-hostname-nslookup-fails-with-bad-address

wolph commented 4 years ago

After looking around a bit I think the only "good" solution is to leave the /etc/resolv.conf alone in kubernetes and fix it by modifying the spec.dnsConfig settings.

I don't think that's possible right now though. It's either writing the local dns server when unbound is enabled or 1.1.1.1 if not. Setting DNS_PLAINTEXT_ADDRESS='' doesn't work either.

qdm12 commented 4 years ago

Lucky us, specifying multiple nameservers in /etc/resolv.conf works by order. It will try the first one and then the second one etc. on failure.

I'll therefore add whatever DNS before the already existing nameserver instead of overwriting it. That should do it. I'll do that tonight and we can then test.

qdm12 commented 4 years ago

Let me know if it works for you! There is also a rather big firewall refactor incoming tonight/tomorrow, hopefully I won't break anything 😄 I'll make a release in the coming days as well when it's proven enough stable.

wolph commented 4 years ago

So far it's not working unfortunately... I'm investigating what's causing it

qdm12 commented 4 years ago

Check if /etc/resolv.conf is as expected (I haven't, shame on me!)

wolph commented 4 years ago

It is. I think it's a routing issue due to the vpn actually

# nslookup kubernetes.default.svc.cluster.local 10.245.0.10
;; connection timed out; no servers could be reached

It's not making sense to me though... I'm trying to get a traceroute app installed so I can test

wolph commented 4 years ago

Well.. I got a little further, the firewall was blocking me. After that (and changing the order in /etc/resolv.conf) it works

In either case... if the kubernetes resolver is not at the top it doesn't work.

wolph commented 4 years ago

While changing the order is an option, I think the better solution would be to specify the k8s dns server as the upstream for svc.cluster.local

For now, either would work though :)

qdm12 commented 4 years ago

Actually I have to re-change the logic for writing to /etc/resolv.conf because dns blocking won't work with the way it is. I'll probably add an env variable for keeping the other nameservers.

What order did you use in the end in /etc/resolv.conf?
Do you mean to add i.e. nameserver svc.cluster.local to /etc/resolv.conf? That could be an env variable as well.

wolph commented 4 years ago

The k8s dns server needs to be first, otherwise it doesn’t seem to work
I mean modifying the unbound configuration to use the k8s dns server for the k8s addresses. So effectively creating a stub zone only for svc.cluster.local (or .local entirely)

wolph commented 4 years ago

I think it should look something like this:

stub-zone:
  name: "svc.cluster.local"
  stub-addr: 10.245.0.10

qdm12 commented 4 years ago

Would there be a way to set it somehow directly in /etc/resolv.conf instead? Can the container know the stub address automatically (10.245.0.10) maybe?

I also just pushed a lot of changes, hopefully nothing should break but let me know 😉 I also put back resolv.conf the way it was before as mentioned above.

wolph commented 4 years ago

Having the k8s dns first at least makes it work. But I'm not sure if it only resolves k8s addresses that way or if it resolves all hostnames through the k8s dns server.

If it does use k8s for all dns that might mean your dns requests leak outside of the vpn connection. With location based dns servers that could be problematic

wolph commented 4 years ago

As for knowing the stub addres... I'm not sure unfortunately. I'm not that well versed with k8s dns yet. I think it's supposed to be passed through the --cluster-dns=... parameter but I'm not sure how/where to extract that: https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#introduction

wolph commented 4 years ago

@qdm12 for the time being, would it be possible to put the custom dns server after the k8s dns server in /etc/resolv.conf? That way it at least works for both :)

qdm12 commented 4 years ago

Try with DNS_KEEP_NAMESERVER=on it should only append the nameserver to the bottom of /etc/resolv.conf. Although the latest image might still be building.

wolph commented 4 years ago

Oops... tested from the wrong server. Guess it's still building. I'll retry again in a few minutes

qdm12 commented 4 years ago

Yes, sorry the build has been failing... It's still building now (here) or wait for green mark instead of that orange dot next to the commit referenced above.

wolph commented 4 years ago

For some reason I'm still having issues... it's definitely a routing thing but manually trying to fix it does very little.

The default routes when the VPN is connected:

/ # route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.8.8.1        128.0.0.0       UG    0      0        0 tun0
0.0.0.0         10.244.1.32     0.0.0.0         UG    0      0        0 eth0
10.8.8.0        0.0.0.0         255.255.255.0   U     0      0        0 tun0
10.244.1.32     0.0.0.0         255.255.255.255 UH    0      0        0 eth0
89.187.174.231  10.244.1.32     255.255.255.255 UGH   0      0        0 eth0
128.0.0.0       10.8.8.1        128.0.0.0       UG    0      0        0 tun0

Which enables internet connectivity without any issues. Pinging the k8s gateway works too. But pinging any device (including the dns server) doesn't :/

/ # traceroute -nv 10.245.0.10
traceroute to 10.245.0.10 (10.245.0.10), 30 hops max, 46 byte packets
 1  10.8.8.1 54 bytes to (null)  1581.624 ms  1.266 ms  1.251 ms
 2  89.187.174.254 54 bytes to (null)  1.307 ms  1.366 ms  1.335 ms

So I figured, let's add a route for that:

/ # route add -net 10.245.0.0/16 gw 10.244.1.32
/ # traceroute -nv 10.245.0.10
traceroute to 10.245.0.10 (10.245.0.10), 30 hops max, 46 byte packets
 1  *  *  *
 2  10.244.1.32 54 bytes to (null)  0.006 ms  0.003 ms  0.002 ms
 3  161.35.144.254 36 bytes to (null)  2.216 ms  161.35.144.253 36 bytes to (null)  2.410 ms  161.35.144.254 36 bytes to (null)  2.704 ms

Oddly enough... still no dice. I'm probably forgetting about something here or it's because the ranges are overlapping but I'm not getting a connection to k8s yet.

wolph commented 4 years ago

I found the solution! I forgot to add the device when adding the route. If I manually add the route it works now:

ip route add 10.192.0.0/10 via 10.244.1.32 dev eth0

Would it be possible to add a custom script to run after the connection has been set up?

qdm12 commented 4 years ago

A few questions:

Do you need the nameserver thing or does this routing solves it by itself?
Is 10.244.1.32 the default gateway? (i.e. for me i get in the logs routing: default route found: interface tun0, gateway 10.3.10.5, subnet 0.0.0.0/0) - by the way the 0.0.0.0/0 is wrong, I need to fix it.

wolph commented 4 years ago

Both are needed, otherwise it can't resolve local addresses anymore.
Yes, it's the default route but I think it's due to overlapping ip ranges because the vpn takes precedence.

qdm12 commented 4 years ago

Ok let's keep the DNS as it is now with the env variable
For now, the program does ip route add <some-subnet> via <defaultgateway> dev <defaultinterface> for each subnet specified in EXTRA_SUBNETS (as well as some iptables accepting). eth0 should be your default interface and 10.3.10.5 your default gateway, so maybe specifying EXTRA_SUBNETS=10.192.0.0/10 would work?

On a side note, maybe that local subnet being detected as 0.0.0.0/0 was messing everything up, it's fixed in the commit mentioned above, it was probably introduced when switching to Alpine 3.12. Try again with that version and see what local subnet it finds. If 10.192.0.0/10 is actually the local subnet the container is running it, it should automatically work (with that fix).

qdm12 commented 4 years ago

Also there was a bug the last 1-3 days where the EXTRA_SUBNETS would just be ignored, which is fixed in #190 so that may had affected your testing.

wolph commented 4 years ago

I'm not seeing any difference unfortunately. The routing table looks identical, am I using it incorrectly?

=========================================
================ Gluetun ================
=========================================
==== A mix of OpenVPN, DNS over TLS, ====
======= Shadowsocks and Tinyproxy =======
========= all glued up with Go ==========
=========================================
=========== For tunneling to ============
======== your favorite VPN server =======
=========================================
=== Made with ❤️  by github.com/qdm12 ====
=========================================

Running version latest built on 2020-07-12T19:23:29Z (commit 0731b1c)

🔧  Need help? https://github.com/qdm12/private-internet-access-docker/issues/new
💻  Email? quentin.mcgaw@gmail.com
☕  Slack? Join from the Slack button on Github
💸  Help me? https://github.com/sponsors/qdm12
2020-07-12T20:56:54.052Z        INFO    OpenVPN version: 2.4.9
2020-07-12T20:56:54.054Z        INFO    Unbound version: 1.10.1
2020-07-12T20:56:54.057Z        INFO    IPtables version: v1.8.4
2020-07-12T20:56:54.078Z        INFO    TinyProxy version: 1.10.0
2020-07-12T20:56:54.080Z        INFO    ShadowSocks version: 3.3.4
2020-07-12T20:56:54.081Z        INFO    Settings summary below:
OpenVPN settings:
|--User: [redacted]
|--Password: [redacted]
|--Verbosity level: 1
|--Run as root: no
|--Surfshark settings:
 |--Network protocol: udp
 |--Region: netherlands amsterdam
System settings:
|--User ID: 1000
|--Group ID: 1000
|--Timezone:
|--IP Status filepath: /ip
DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: disabled
 |--Block malicious: disabled
 |--Block surveillance: disabled
 |--Block ads: disabled
 |--Allowed hostnames:
  |--
 |--Private addresses:
  |--127.0.0.1/8
  |--10.0.0.0/8
  |--172.16.0.0/12
  |--192.168.0.0/16
  |--169.254.0.0/16
  |--::1/128
  |--fc00::/7
  |--fe80::/10
  |--::ffff:0:0/96
 |--Verbosity level: 1/5
 |--Verbosity details level: 0/4
 |--Validation log level: 0/2
 |--IPv6 resolution: disabled
 |--Update: deactivated
 |--Keep nameserver (disabled blocking): yes
Firewall settings:
 |--Allowed subnets: 10.192.0.0/10
TinyProxy settings:
Port: 8888
 |--Authentication: disabled
 |--Log level: info
ShadowSocks settings: disabled

2020-07-12T20:56:54.081Z        INFO    openvpn configurator: checking for device /dev/net/tun
2020-07-12T20:56:54.082Z        INFO    firewall: enabling...
2020-07-12T20:56:54.082Z        INFO    Launching standard output merger
2020-07-12T20:56:54.082Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T20:56:54.082Z        INFO    routing: local subnet found: 10.244.1.32/32
2020-07-12T20:56:54.093Z        INFO    firewall: enabled successfully
2020-07-12T20:56:54.094Z        INFO    tinyproxy configurator: generating tinyproxy configuration file
2020-07-12T20:56:54.094Z        INFO    http server: listening on 0.0.0.0:8000
2020-07-12T20:56:54.094Z        INFO    firewall: setting VPN connections through firewall...
2020-07-12T20:56:54.094Z        INFO    dns over tls: falling back on plaintext DNS at address 1.1.1.1
2020-07-12T20:56:54.094Z        INFO    dns configurator: using DNS address 1.1.1.1 internally
2020-07-12T20:56:54.095Z        INFO    dns configurator: using DNS address 1.1.1.1 system wide
2020-07-12T20:56:54.095Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T20:56:54.123Z        INFO    openvpn configurator: starting openvpn
2020-07-12T20:56:54.123Z        INFO    firewall: setting allowed port 8888 through firewall...
2020-07-12T20:56:54.125Z        INFO    openvpn: OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
2020-07-12T20:56:54.125Z        INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-07-12T20:56:54.126Z        INFO    openvpn: WARNING: you are using user/group/chroot/setcon without persist-tun -- this may cause restarts to fail
2020-07-12T20:56:54.129Z        INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]185.59.222.92:1194
2020-07-12T20:56:54.129Z        INFO    openvpn: UDP link local: (not bound)
2020-07-12T20:56:54.129Z        INFO    openvpn: UDP link remote: [AF_INET]185.59.222.92:1194
2020-07-12T20:56:54.129Z        INFO    openvpn: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-07-12T20:56:54.131Z        INFO    tinyproxy configurator: starting tinyproxy server
2020-07-12T20:56:54.145Z        INFO    openvpn: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1633', remote='link-mtu 1581'
2020-07-12T20:56:54.145Z        INFO    openvpn: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
2020-07-12T20:56:54.145Z        INFO    openvpn: WARNING: 'auth' is used inconsistently, local='auth SHA512', remote='auth [null-digest]'
2020-07-12T20:56:54.145Z        INFO    openvpn: [nl-ams-v031.prod.surfshark.com] Peer Connection Initiated with [AF_INET]185.59.222.92:1194
2020-07-12T20:56:54.157Z        INFO    tinyproxy: INFO Initializing tinyproxy ...
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO Reloading config file
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO Disabling transmission of the "Via" header.
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO listen_sock called with addr = '(NULL)'
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO trying to listen on host[0.0.0.0], family[2], socktype[1], proto[6]
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO listening on fd [4]
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO trying to listen on host[::], family[10], socktype[1], proto[6]
2020-07-12T20:56:54.158Z        INFO    tinyproxy: INFO listening on fd [5]
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Now running as group "1000".
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Now running as user "1000".
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 1 of 10 ...
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 2 of 10 ...
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 3 of 10 ...
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 4 of 10 ...
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 5 of 10 ...
2020-07-12T20:56:54.159Z        INFO    tinyproxy: INFO Creating child number 6 of 10 ...
2020-07-12T20:56:54.160Z        INFO    tinyproxy: INFO Creating child number 7 of 10 ...
2020-07-12T20:56:54.160Z        INFO    tinyproxy: INFO Creating child number 8 of 10 ...
2020-07-12T20:56:54.160Z        INFO    tinyproxy: INFO Creating child number 9 of 10 ...
2020-07-12T20:56:54.160Z        INFO    tinyproxy: INFO Creating child number 10 of 10 ...
2020-07-12T20:56:54.160Z        INFO    tinyproxy: INFO Finished creating all children.
2020-07-12T20:56:54.161Z        INFO    tinyproxy: INFO Setting the various signals.
2020-07-12T20:56:54.161Z        INFO    tinyproxy: INFO Starting main loop. Accepting connections.
2020-07-12T20:56:55.185Z        INFO    openvpn: Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:7: block-outside-dns (2.4.9)
2020-07-12T20:56:55.186Z        INFO    openvpn: TUN/TAP device tun0 opened
2020-07-12T20:56:55.186Z        INFO    openvpn: /sbin/ip link set dev tun0 up mtu 1500
2020-07-12T20:56:55.187Z        INFO    openvpn: /sbin/ip addr add dev tun0 10.8.8.19/24 broadcast 10.8.8.255
2020-07-12T20:56:55.190Z        INFO    openvpn: UID set to nonrootuser
2020-07-12T20:56:55.190Z        INFO    openvpn: Initialization Sequence Completed
2020-07-12T20:56:55.190Z        INFO    dns configurator: downloading root hints from https://raw.githubusercontent.com/qdm12/files/master/named.root.updated
2020-07-12T20:56:55.190Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T20:56:55.190Z        INFO    Gateway VPN IP address: 185.59.222.92
2020-07-12T20:56:55.458Z        INFO    ip getter: Public IP address is 185.59.222.93
2020-07-12T20:56:55.530Z        INFO    dns configurator: downloading root key from https://raw.githubusercontent.com/qdm12/files/master/root.key.updated
2020-07-12T20:56:55.707Z        INFO    dns configurator: generating Unbound configuration
2020-07-12T20:56:55.707Z        INFO    dns configurator: 0 hostnames blocked overall
2020-07-12T20:56:55.707Z        INFO    dns configurator: 9 IP addresses blocked overall
2020-07-12T20:56:55.708Z        INFO    dns configurator: starting unbound
2020-07-12T20:56:55.708Z        INFO    dns configurator: using DNS address 127.0.0.1 internally
2020-07-12T20:56:55.708Z        INFO    dns configurator: using DNS address 127.0.0.1 system wide
2020-07-12T20:56:55.723Z        INFO    unbound: notice: init module 0: validator
2020-07-12T20:56:55.724Z        INFO    unbound: notice: init module 1: iterator
2020-07-12T20:56:55.757Z        INFO    unbound: info: start of service (unbound 1.10.1).
2020-07-12T20:56:56.031Z        INFO    unbound: info: generate keytag query _ta-4a5c-4f66. NULL IN

wolph commented 4 years ago

Little question... is the route being added after connecting to the VPN or before? I think it can only work if it's being added after being connected every time. The route could also be added to the ovpn file.

qdm12 commented 4 years ago

It says your local subnet is your default gateway routing: local subnet found: 10.244.1.32/32. In my standard Docker thing, it gives me the Docker network subnet i.e. 172.17.0.0/16. Can you try to run a container and send the output from ip route and the output from cat cat /proc/net/route? Maybe my routing parsing goes wrong with your setup, I'll try to fix it. It would help me understand a bit what's going on too.
The routes for each extra subnets are added before the vpn is connected, but it's easy to change that. Although I don't believe it would matter (correct me if I'm wrong)

wolph commented 4 years ago

Yeah, here you go :)

/ # cat /proc/net/route
Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask            MTU     Window  IRTT
tun0    00000000        0108080A        0003    0       0       0       00000080        0       0       0
eth0    00000000        2001F40A        0003    0       0       0       00000000        0       0       0
tun0    0008080A        00000000        0001    0       0       0       00FFFFFF        0       0       0
eth0    2001F40A        00000000        0005    0       0       0       FFFFFFFF        0       0       0
tun0    00000080        0108080A        0003    0       0       0       00000080        0       0       0
eth0    5CDE3BB9        2001F40A        0007    0       0       0       FFFFFFFF        0       0       0

/ # ip route
0.0.0.0/1 via 10.8.8.1 dev tun0
default via 10.244.1.32 dev eth0 mtu 1450
10.8.8.0/24 dev tun0 proto kernel scope link src 10.8.8.19
10.244.1.32 dev eth0 scope link
128.0.0.0/1 via 10.8.8.1 dev tun0
185.59.222.92 via 10.244.1.32 dev eth0

As far as I have experienced so far the last route added takes precedence, unless the metric parameter is given of course. So I do think it matters to only add it after the VPN has connected

qdm12 commented 4 years ago

Also the fix I just mentioned about EXTRA_SUBNETS I pushed it to the wrong branch, I just pushed it to master now, sorry about that. So it should show in the latest docker image soon. (see #190). Gonna go eat, will check back those routes soon. Thanks!

EDIT: That 0.0.0.0/1 is an error, fixed a few seconds ago.

wolph commented 4 years ago

The new results:

/ # cat /proc/net/route
Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask            MTU     Window  IRTT
tun0    00000000        0108080A        0003    0       0       0       00000080        0       0       0
eth0    00000000        2001F40A        0003    0       0       0       00000000        0       0       0
tun0    0008080A        00000000        0001    0       0       0       00FFFFFF        0       0       0
eth0    2001F40A        00000000        0005    0       0       0       FFFFFFFF        0       0       0
tun0    00000080        0108080A        0003    0       0       0       00000080        0       0       0
eth0    5CDE3BB9        2001F40A        0007    0       0       0       FFFFFFFF        0       0       0
/ #
/ # %
(likegrowers-webliker) rick@lappie# kubectl exec --stdin --tty -n default surfshark-netherlands-000 -c proxy -- /bin/sh
/ # cat /proc/net/route
Iface   Destination     Gateway         Flags   RefCnt  Use     Metric  Mask            MTU     Window  IRTT
tun0    00000000        0108080A        0003    0       0       0       00000080        0       0       0
eth0    00000000        2001F40A        0003    0       0       0       00000000        0       0       0
tun0    0008080A        00000000        0001    0       0       0       00FFFFFF        0       0       0
eth0    0000C00A        2001F40A        0003    0       0       0       0000C0FF        0       0       0
eth0    2001F40A        00000000        0005    0       0       0       FFFFFFFF        0       0       0
tun0    00000080        0108080A        0003    0       0       0       00000080        0       0       0
eth0    C22366D4        2001F40A        0007    0       0       0       FFFFFFFF        0       0       0
/ # route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.8.8.1        128.0.0.0       UG    0      0        0 tun0
default         10.244.1.32     0.0.0.0         UG    0      0        0 eth0
10.8.8.0        *               255.255.255.0   U     0      0        0 tun0
10.192.0.0      10.244.1.32     255.192.0.0     UG    0      0        0 eth0
10.244.1.32     *               255.255.255.255 UH    0      0        0 eth0
128.0.0.0       10.8.8.1        128.0.0.0       UG    0      0        0 tun0
212.102.35.194  10.244.1.32     255.255.255.255 UGH   0      0        0 eth0
/ # ip route
0.0.0.0/1 via 10.8.8.1 dev tun0
default via 10.244.1.32 dev eth0 mtu 1450
10.8.8.0/24 dev tun0 proto kernel scope link src 10.8.8.29
10.192.0.0/10 via 10.244.1.32 dev eth0
10.244.1.32 dev eth0 scope link
128.0.0.0/1 via 10.8.8.1 dev tun0
212.102.35.194 via 10.244.1.32 dev eth0

And the new logs:

=========================================
================ Gluetun ================
=========================================
==== A mix of OpenVPN, DNS over TLS, ====
======= Shadowsocks and Tinyproxy =======
========= all glued up with Go ==========
=========================================
=========== For tunneling to ============
======== your favorite VPN server =======
=========================================
=== Made with ❤️  by github.com/qdm12 ====
=========================================

Running version latest built on 2020-07-12T21:22:24Z (commit 4cd6b33)

🔧  Need help? https://github.com/qdm12/private-internet-access-docker/issues/new
💻   Email? quentin.mcgaw@gmail.com
☕   Slack? Join from the Slack button on Github
💸   Help me? https://github.com/sponsors/qdm12
2020-07-12T21:36:06.482Z        INFO    Unbound version: 1.10.1
2020-07-12T21:36:06.483Z        INFO    IPtables version: v1.8.4
2020-07-12T21:36:06.514Z        INFO    TinyProxy version: 1.10.0
2020-07-12T21:36:06.526Z        INFO    ShadowSocks version: 3.3.4
2020-07-12T21:36:06.536Z        INFO    OpenVPN version: 2.4.9
2020-07-12T21:36:06.536Z        INFO    Settings summary below:
OpenVPN settings:
|--User: [redacted]
|--Password: [redacted]
|--Verbosity level: 1
|--Run as root: no
|--Surfshark settings:
 |--Network protocol: udp
 |--Region: netherlands amsterdam
System settings:
|--User ID: 1000
|--Group ID: 1000
|--Timezone:
|--IP Status filepath: /ip
DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: disabled
 |--Block malicious: disabled
 |--Block surveillance: disabled
 |--Block ads: disabled
 |--Allowed hostnames:
  |--
 |--Private addresses:
  |--127.0.0.1/8
  |--10.0.0.0/8
  |--172.16.0.0/12
  |--192.168.0.0/16
  |--169.254.0.0/16
  |--::1/128
  |--fc00::/7
  |--fe80::/10
  |--::ffff:0:0/96
 |--Verbosity level: 1/5
 |--Verbosity details level: 0/4
 |--Validation log level: 0/2
 |--IPv6 resolution: disabled
 |--Update: deactivated
 |--Keep nameserver (disabled blocking): yes
Firewall settings:
 |--Allowed subnets: 10.192.0.0/10
TinyProxy settings:
Port: 8888
 |--Authentication: disabled
 |--Log level: info
ShadowSocks settings: disabled

2020-07-12T21:36:06.537Z        INFO    openvpn configurator: checking for device /dev/net/tun
2020-07-12T21:36:06.537Z        INFO    firewall: enabling...
2020-07-12T21:36:06.537Z        INFO    Launching standard output merger
2020-07-12T21:36:06.537Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T21:36:06.537Z        INFO    routing: local subnet found: 10.244.1.32/32
2020-07-12T21:36:06.576Z        INFO    firewall: enabled successfully
2020-07-12T21:36:06.576Z        INFO    firewall: setting allowed subnets through firewall...
2020-07-12T21:36:06.576Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T21:36:06.578Z        INFO    routing: adding 10.192.0.0/10 as route via 10.244.1.32 eth0
2020-07-12T21:36:06.580Z        INFO    dns over tls: falling back on plaintext DNS at address 1.1.1.1
2020-07-12T21:36:06.580Z        INFO    dns configurator: using DNS address 1.1.1.1 internally
2020-07-12T21:36:06.580Z        INFO    dns configurator: using DNS address 1.1.1.1 system wide
2020-07-12T21:36:06.581Z        INFO    tinyproxy configurator: generating tinyproxy configuration file
2020-07-12T21:36:06.581Z        INFO    firewall: setting allowed port 8888 through firewall...
2020-07-12T21:36:06.581Z        INFO    http server: listening on 0.0.0.0:8000
2020-07-12T21:36:06.583Z        INFO    tinyproxy configurator: starting tinyproxy server
2020-07-12T21:36:06.583Z        INFO    firewall: setting VPN connections through firewall...
2020-07-12T21:36:06.584Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T21:36:06.610Z        INFO    tinyproxy: INFO Initializing tinyproxy ...
2020-07-12T21:36:06.611Z        INFO    tinyproxy: INFO Reloading config file
2020-07-12T21:36:06.611Z        INFO    tinyproxy: INFO Disabling transmission of the "Via" header.
2020-07-12T21:36:06.611Z        INFO    tinyproxy: INFO listen_sock called with addr = '(NULL)'
2020-07-12T21:36:06.611Z        INFO    tinyproxy: INFO trying to listen on host[0.0.0.0], family[2], socktype[1], proto[6]
2020-07-12T21:36:06.611Z        INFO    tinyproxy: INFO listening on fd [4]
2020-07-12T21:36:06.612Z        INFO    tinyproxy: INFO trying to listen on host[::], family[10], socktype[1], proto[6]
2020-07-12T21:36:06.612Z        INFO    tinyproxy: INFO listening on fd [5]
2020-07-12T21:36:06.612Z        INFO    tinyproxy: INFO Now running as group "1000".
2020-07-12T21:36:06.612Z        INFO    tinyproxy: INFO Now running as user "1000".
2020-07-12T21:36:06.613Z        INFO    tinyproxy: INFO Creating child number 1 of 10 ...
2020-07-12T21:36:06.614Z        INFO    tinyproxy: INFO Creating child number 2 of 10 ...
2020-07-12T21:36:06.614Z        INFO    tinyproxy: INFO Creating child number 3 of 10 ...
2020-07-12T21:36:06.615Z        INFO    tinyproxy: INFO Creating child number 4 of 10 ...
2020-07-12T21:36:06.615Z        INFO    tinyproxy: INFO Creating child number 5 of 10 ...
2020-07-12T21:36:06.616Z        INFO    tinyproxy: INFO Creating child number 6 of 10 ...
2020-07-12T21:36:06.616Z        INFO    tinyproxy: INFO Creating child number 7 of 10 ...
2020-07-12T21:36:06.616Z        INFO    tinyproxy: INFO Creating child number 8 of 10 ...
2020-07-12T21:36:06.617Z        INFO    tinyproxy: INFO Creating child number 9 of 10 ...
2020-07-12T21:36:06.617Z        INFO    tinyproxy: INFO Creating child number 10 of 10 ...
2020-07-12T21:36:06.617Z        INFO    tinyproxy: INFO Finished creating all children.
2020-07-12T21:36:06.618Z        INFO    tinyproxy: INFO Setting the various signals.
2020-07-12T21:36:06.618Z        INFO    tinyproxy: INFO Starting main loop. Accepting connections.
2020-07-12T21:36:06.624Z        INFO    openvpn configurator: starting openvpn
2020-07-12T21:36:06.626Z        INFO    openvpn: OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
2020-07-12T21:36:06.626Z        INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-07-12T21:36:06.627Z        INFO    openvpn: WARNING: you are using user/group/chroot/setcon without persist-tun -- this may cause restarts to fail
2020-07-12T21:36:06.629Z        INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]212.102.35.194:1194
2020-07-12T21:36:06.629Z        INFO    openvpn: UDP link local: (not bound)
2020-07-12T21:36:06.629Z        INFO    openvpn: UDP link remote: [AF_INET]212.102.35.194:1194
2020-07-12T21:36:06.629Z        INFO    openvpn: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-07-12T21:36:06.640Z        INFO    openvpn: WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1633', remote='link-mtu 1581'
2020-07-12T21:36:06.640Z        INFO    openvpn: WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
2020-07-12T21:36:06.640Z        INFO    openvpn: WARNING: 'auth' is used inconsistently, local='auth SHA512', remote='auth [null-digest]'
2020-07-12T21:36:06.640Z        INFO    openvpn: [nl-ams-v063.prod.surfshark.com] Peer Connection Initiated with [AF_INET]212.102.35.194:1194
2020-07-12T21:36:07.775Z        INFO    openvpn: Options error: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:7: block-outside-dns (2.4.9)
2020-07-12T21:36:07.776Z        INFO    openvpn: TUN/TAP device tun0 opened
2020-07-12T21:36:07.776Z        INFO    openvpn: /sbin/ip link set dev tun0 up mtu 1500
2020-07-12T21:36:07.777Z        INFO    openvpn: /sbin/ip addr add dev tun0 10.8.8.29/24 broadcast 10.8.8.255
2020-07-12T21:36:07.780Z        INFO    openvpn: UID set to nonrootuser
2020-07-12T21:36:07.780Z        INFO    openvpn: Initialization Sequence Completed
2020-07-12T21:36:07.780Z        INFO    dns configurator: downloading root hints from https://raw.githubusercontent.com/qdm12/files/master/named.root.updated
2020-07-12T21:36:07.780Z        INFO    routing: default route found: interface eth0, gateway 10.244.1.32
2020-07-12T21:36:07.780Z        INFO    Gateway VPN IP address: 212.102.35.194
2020-07-12T21:36:07.917Z        INFO    dns configurator: downloading root key from https://raw.githubusercontent.com/qdm12/files/master/root.key.updated
2020-07-12T21:36:07.921Z        INFO    dns configurator: generating Unbound configuration
2020-07-12T21:36:07.922Z        INFO    dns configurator: 0 hostnames blocked overall
2020-07-12T21:36:07.922Z        INFO    dns configurator: 9 IP addresses blocked overall
2020-07-12T21:36:07.922Z        INFO    dns configurator: starting unbound
2020-07-12T21:36:07.922Z        INFO    dns configurator: using DNS address 127.0.0.1 internally
2020-07-12T21:36:07.923Z        INFO    dns configurator: using DNS address 127.0.0.1 system wide
2020-07-12T21:36:07.937Z        INFO    unbound: notice: init module 0: validator
2020-07-12T21:36:07.938Z        INFO    unbound: notice: init module 1: iterator
2020-07-12T21:36:07.980Z        INFO    unbound: info: start of service (unbound 1.10.1).
2020-07-12T21:36:08.043Z        INFO    ip getter: Public IP address is 212.102.35.195
2020-07-12T21:36:08.241Z        INFO    unbound: info: generate keytag query _ta-4a5c-4f66. NULL IN

wolph commented 4 years ago

Oddly enough, manually removing and adding the route doesn't help either now. I can't ping anything besides the k8s gateway

wolph commented 4 years ago

I should note that when using the machine as a separate k8s pod with tinyproxy as a service it does work now. It's only as a sidecar that it doesn't work

qdm12 commented 4 years ago

There is actually an issue still (#194) which might be related. Sorry about that, I'll fix that one first and would probably ask you to test again then once it's fixed. Have a good week!

wolph commented 4 years ago

Thanks for all the help!

For now I'll just run the VPN as separate deployments and I'll try the sidecar option again in the future :)

qdm12 commented 4 years ago

Alright, found/pushed the fix! You could try now (or anytime later, no pressure). The routes were setup correctly but the firewall would be blocking everything. Not sure it's very relevant to your situation, but perhaps.

wolph commented 4 years ago

I had the firewall disabled to make sure I didn't have any issues with it. But enabling it immediately disallows pinging the k8s IP addresses.

The routing table looks a bit odd to me:

/ # ip route show table local
broadcast 10.8.8.0 dev tun0 proto kernel scope link src 10.8.8.3
local 10.8.8.3 dev tun0 proto kernel scope host src 10.8.8.3
broadcast 10.8.8.255 dev tun0 proto kernel scope link src 10.8.8.3
local 10.244.2.29 dev eth0 proto kernel scope host src 10.244.2.29
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
/ # ip route show table main
0.0.0.0/1 via 10.8.8.1 dev tun0
default via 10.244.2.110 dev eth0 mtu 1450
10.8.8.0/24 dev tun0 proto kernel scope link src 10.8.8.3
10.128.0.0/9 via 10.244.2.110 dev eth0
10.244.2.110 dev eth0 scope link
89.46.223.214 via 10.244.2.110 dev eth0
128.0.0.0/1 via 10.8.8.1 dev tun0

But... it's working! For some reason I can't ping the k8s dns server anymore but that's not working from other machines either... no clue how/why but it looks like k8s changed something. But it does work! :D

I do think something is wrong with the firewall system though as it's blocking everything when enabled. The current firewall rules:

/ # iptables -vnL
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   76  6480 ACCEPT     all  --  lo     *       0.0.0.0/0            0.0.0.0/0
 4574 2170K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  *      *       10.244.2.110         10.244.2.110
    0     0 ACCEPT     all  --  eth0   *       10.128.0.0/9         10.244.2.110
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:8888
    0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:8888

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy DROP 8 packets, 772 bytes)
 pkts bytes target     prot opt in     out     source               destination
   76  6480 ACCEPT     all  --  *      lo      0.0.0.0/0            0.0.0.0/0
 4667  527K ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
  226 14244 ACCEPT     all  --  *      tun0    0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       10.244.2.110         10.244.2.110
    0     0 ACCEPT     all  --  *      eth0    10.244.2.110         10.128.0.0/9
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.208.56         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.208.66         udp dpt:1194
    1   114 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.208.68         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.209.20         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.209.57         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.209.113        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.209.120        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            81.19.209.124        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.72         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.78         udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.104        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.212        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.214        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.217        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.222        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.46.223.229        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.187.174.229       udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            89.187.174.231       udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.59.222.92        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.59.222.94        udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.59.222.166       udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            185.59.222.168       udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            212.102.35.194       udp dpt:1194
    0     0 ACCEPT     udp  --  *      eth0    0.0.0.0/0            212.102.35.196       udp dpt:1194
/ # ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.8.8.4/24 brd 10.8.8.255 scope global tun0
       valid_lft forever preferred_lft forever
240: eth0@if241: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether fa:16:d3:cb:99:04 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.244.2.122/32 scope global eth0
       valid_lft forever preferred_lft forever

The firewall rules look wrong. It's filtering on 10.244.2.110 as a source IP which is the gateway so that won't work. But with the firewall disabled everything works beyond that. Thanks for all the help!

qdm12 commented 4 years ago

Hello, sorry I lost track a bit. Let's find how to make it work with the firewall on. Keeping it off disables the kill switch and might be dangerous i.e. vpn provider misbehaving and trying to reach your system through the tunnel.

Just repeating myself, but to be sure:

what's the local subnet shown in the logs of gluetun?
what's the k8s ip addresses you would need to ping?

And to be sure, starting the container with the firewall on just fails straight away right? Thanks!

wolph commented 3 years ago

Took a while for me to get back to this... it's been quite a busy period

Anyhow, here's the log:

=========================================
================ Gluetun ================
=========================================
==== A mix of OpenVPN, DNS over TLS, ====
======= Shadowsocks and Tinyproxy =======
========= all glued up with Go ==========
=========================================
=========== For tunneling to ============
======== your favorite VPN server =======
=========================================
=== Made with ??  by github.com/qdm12 ====
=========================================

Running version latest built on 2020-09-05T16:57:43Z (commit 797fa33)

??  Persistent server IP addresses at /gluetun/servers.json, please BIND MOUNT

??  Need help? https://github.com/qdm12/gluetun/issues/new
??  Email? quentin.mcgaw@gmail.com
?  Slack? Join from the Slack button on Github
??  Help me? https://github.com/sponsors/qdm12
2020-09-10T14:37:44.416Z        INFO    TinyProxy version: 1.10.0
2020-09-10T14:37:44.427Z        INFO    OpenVPN version: 2.4.9
2020-09-10T14:37:44.431Z        INFO    Unbound version: 1.10.1
2020-09-10T14:37:44.454Z        INFO    IPtables version: v1.8.4
2020-09-10T14:37:44.454Z        INFO    Settings summary below:
OpenVPN settings:
|--User: [redacted]
|--Password: [redacted]
|--Verbosity level: 1
|--Run as root: no
|--Surfshark settings:
 |--Network protocol: udp
 |--Region: us los angeles
System settings:
|--User ID: 1000
|--Group ID: 1000
|--Timezone:
|--IP Status filepath: /tmp/gluetun/ip
DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: disabled
 |--Block malicious: disabled
 |--Block surveillance: disabled
 |--Block ads: disabled
 |--Allowed hostnames:
  |--
 |--Private addresses:
  |--127.0.0.1/8
  |--10.0.0.0/8
  |--172.16.0.0/12
  |--192.168.0.0/16
  |--169.254.0.0/16
  |--::1/128
  |--fc00::/7
  |--fe80::/10
  |--::ffff:0:0/96
 |--Verbosity level: 1/5
 |--Verbosity details level: 0/4
 |--Validation log level: 0/2
 |--IPv6 resolution: disabled
 |--Update: deactivated
 |--Keep nameserver (disabled blocking): yes
Firewall settings: disabled
TinyProxy settings:
Port: 8888
 |--Authentication: disabled
 |--Log level: info
ShadowSocks settings:
 |--Password: [redacted]
 |--Log: disabled
 |--Port: 8388
 |--Method: chacha20-ietf-poly1305
Public IP check period: 12h0m0s
Version information: enabled

2020-09-10T14:37:44.455Z        INFO    storage: Merging by most recent 6355 hardcoded servers and 0 servers read from /gluetun/servers.json
2020-09-10T14:37:44.507Z        INFO    routing: default route found: interface eth0, gateway 10.244.4.152
2020-09-10T14:37:44.507Z        INFO    routing: local subnet found: 10.244.4.152/32
2020-09-10T14:37:44.507Z        INFO    openvpn configurator: checking for device /dev/net/tun
2020-09-10T14:37:44.507Z        INFO    firewall: firewall disabled, only updating allowed subnets internal list and updating routes
2020-09-10T14:37:44.507Z        INFO    routing: adding 10.0.0.0/8 as route via 10.244.4.152 eth0
2020-09-10T14:37:44.511Z        INFO    Launching standard output merger
2020-09-10T14:37:44.511Z        INFO    dns over tls: falling back on plaintext DNS at address 1.1.1.1
2020-09-10T14:37:44.511Z        INFO    dns configurator: using DNS address 1.1.1.1 internally
2020-09-10T14:37:44.511Z        INFO    dns configurator: using DNS address 1.1.1.1 system wide
2020-09-10T14:37:44.511Z        INFO    tinyproxy configurator: generating tinyproxy configuration file
2020-09-10T14:37:44.512Z        INFO    firewall: firewall disabled, only updating allowed ports internal state
2020-09-10T14:37:44.512Z        INFO    tinyproxy configurator: starting tinyproxy server
2020-09-10T14:37:44.529Z        INFO    firewall: firewall disabled, only updating allowed ports internal state
2020-09-10T14:37:44.530Z        INFO    firewall: firewall disabled, only updating VPN connections internal list
2020-09-10T14:37:44.530Z        INFO    openvpn configurator: starting openvpn
2020-09-10T14:37:44.531Z        INFO    http server: listening on 0.0.0.0:8000
2020-09-10T14:37:44.533Z        INFO    openvpn: OpenVPN 2.4.9 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Apr 20 2020
2020-09-10T14:37:44.534Z        INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-09-10T14:37:44.541Z        INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]192.111.134.205:1194
2020-09-10T14:37:44.541Z        INFO    openvpn: UDP link local: (not bound)
2020-09-10T14:37:44.541Z        INFO    openvpn: UDP link remote: [AF_INET]192.111.134.205:1194
2020-09-10T14:37:44.543Z        INFO    tinyproxy: Initializing tinyproxy ...
2020-09-10T14:37:44.543Z        INFO    tinyproxy: Reloading config file
2020-09-10T14:37:44.544Z        INFO    tinyproxy: Disabling transmission of the "Via" header.
2020-09-10T14:37:44.544Z        INFO    tinyproxy: listen_sock called with addr = '(NULL)'
2020-09-10T14:37:44.544Z        INFO    tinyproxy: trying to listen on host[0.0.0.0], family[2], socktype[1], proto[6]
2020-09-10T14:37:44.544Z        INFO    tinyproxy: listening on fd [4]
2020-09-10T14:37:44.544Z        INFO    tinyproxy: trying to listen on host[::], family[10], socktype[1], proto[6]
2020-09-10T14:37:44.544Z        INFO    tinyproxy: listening on fd [5]
2020-09-10T14:37:44.544Z        INFO    tinyproxy: Now running as group "1000".
2020-09-10T14:37:44.544Z        INFO    tinyproxy: Now running as user "1000".
2020-09-10T14:37:44.545Z        INFO    tinyproxy: Creating child number 1 of 10 ...
2020-09-10T14:37:44.545Z        INFO    tinyproxy: Creating child number 2 of 10 ...
2020-09-10T14:37:44.545Z        INFO    tinyproxy: Creating child number 3 of 10 ...
2020-09-10T14:37:44.545Z        INFO    tinyproxy: Creating child number 4 of 10 ...
2020-09-10T14:37:44.546Z        INFO    tinyproxy: Creating child number 5 of 10 ...
2020-09-10T14:37:44.546Z        INFO    tinyproxy: Creating child number 6 of 10 ...
2020-09-10T14:37:44.546Z        INFO    tinyproxy: Creating child number 7 of 10 ...
2020-09-10T14:37:44.546Z        INFO    tinyproxy: Creating child number 8 of 10 ...
2020-09-10T14:37:44.547Z        INFO    tinyproxy: Creating child number 9 of 10 ...
2020-09-10T14:37:44.547Z        INFO    tinyproxy: Creating child number 10 of 10 ...
2020-09-10T14:37:44.547Z        INFO    tinyproxy: Finished creating all children.
2020-09-10T14:37:44.547Z        INFO    tinyproxy: Setting the various signals.
2020-09-10T14:37:44.547Z        INFO    tinyproxy: Starting main loop. Accepting connections.
2020-09-10T14:37:45.287Z        WARN    openvpn: 'link-mtu' is used inconsistently, local='link-mtu 1633', remote='link-mtu 1581'
2020-09-10T14:37:45.287Z        WARN    openvpn: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
2020-09-10T14:37:45.287Z        WARN    openvpn: 'auth' is used inconsistently, local='auth SHA512', remote='auth [null-digest]'
2020-09-10T14:37:45.287Z        INFO    openvpn: [us-lax-v026.prod.surfshark.com] Peer Connection Initiated with [AF_INET]192.111.134.205:1194
2020-09-10T14:37:46.485Z        ERROR   openvpn: Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:7: block-outside-dns (2.4.9)
2020-09-10T14:37:46.487Z        INFO    openvpn: TUN/TAP device tun0 opened
2020-09-10T14:37:46.487Z        INFO    openvpn: /sbin/ip link set dev tun0 up mtu 1500
2020-09-10T14:37:46.489Z        INFO    openvpn: /sbin/ip addr add dev tun0 10.8.8.2/24 broadcast 10.8.8.255
2020-09-10T14:37:46.512Z        INFO    openvpn: UID set to nonrootuser
2020-09-10T14:37:46.513Z        INFO    openvpn: Initialization Sequence Completed
2020-09-10T14:37:46.513Z        INFO    routing: default route found: interface eth0, gateway 10.244.4.152
2020-09-10T14:37:46.513Z        INFO    Gateway VPN IP address: 192.111.134.205
2020-09-10T14:37:46.513Z        INFO    dns configurator: downloading root hints from https://raw.githubusercontent.com/qdm12/files/master/named.root.updated
2020-09-10T14:37:47.752Z        INFO    ip getter: Public IP address is 192.111.134.206
2020-09-10T14:37:47.841Z        INFO    dns configurator: downloading root key from https://raw.githubusercontent.com/qdm12/files/master/root.key.updated
2020-09-10T14:37:47.939Z        INFO    You are running 3 commits behind the most recent latest
2020-09-10T14:37:47.995Z        INFO    dns configurator: generating Unbound configuration
2020-09-10T14:37:47.995Z        INFO    dns configurator: 0 hostnames blocked overall
2020-09-10T14:37:47.995Z        INFO    dns configurator: 9 IP addresses blocked overall
2020-09-10T14:37:47.996Z        INFO    dns configurator: starting unbound
2020-09-10T14:37:47.996Z        INFO    dns configurator: using DNS address 127.0.0.1 internally
2020-09-10T14:37:47.996Z        INFO    dns configurator: using DNS address 127.0.0.1 system wide
2020-09-10T14:37:48.014Z        INFO    unbound: init module 0: validator
2020-09-10T14:37:48.015Z        INFO    unbound: init module 1: iterator
2020-09-10T14:37:48.050Z        INFO    unbound: start of service (unbound 1.10.1).
2020-09-10T14:37:48.895Z        INFO    unbound: generate keytag query _ta-4a5c-4f66. NULL IN

As for IPs that I need to reach, here's a list of examples:

10.244.2.253
10.244.3.129
10.244.0.140

I think there are also some addresses in the 10.245.0.0/16 range but I can't find any right now

npawelek commented 3 years ago

Nice to see this in the works! I initially had this issue on Kubernetes as well. I ended up using dnsConfig options with DOT disabled. I pointed to an in-cluster pi-hole with DoT enabled to solve the need here.

Let me know if you could use another set of eyes for testing.

qdm12 commented 3 years ago

This is the same as #281 I believe. I'm working on a sort of UDP / DNS proxy to redirect DNS requests to Unbound (DNS over TLS) or the native DNS (from the Docker bridge or K8s) depending on the request. If it has no dot it's sent to the native DNS and otherwise sent to Unbound, such that Unbound block lists can't go through using the native Docker DNS.

Not sure that will solve it for K8s as this one might use dots in its private addresses, but I'll see what I can do. Doing a simple IP check on the result to check it's private could do the trick. Anyway, that'll take several days to finish, but I'll keep you updated. Thanks for your patience!

toniopelo commented 1 year ago

I am stuck on this as well. I got to admit I probably only understood half of the above discussion. What I can report is that playing around with the following env vars didn't help:

DNS_KEEP_NAMESERVER
DOT
UNBLOCK
DNS_ADDRESS
DOT_CACHING
BLOCK_MALICIOUS

I switched everything on and off and didn't use the UNBLOCK in the end. Using DNS_KEEP_NAMESERVER didn't help either, putting DNS_ADDRESS to the original nameserver ip (k8s dns server) is the same. Even manually resetting /etc/resolv.conf to its original value to try if k8s internal dns were resolved didn't help, so I guess there are other stuff to tweak that I don't understand.

I tested all these by executing nslookup someinternalsubdomain.default.svc.cluster.local on the gluetun container and on the main container as well (the container along which gluetun run as a sidecar). In my situation I'm stuck with the sidecar option because my pods (which are jobs) scales in such way that it would be a mess to have it as a separated deployment.

I'm all ears if you have any tip and I can debug further if you can guide me through it @qdm12 :).

PrivatePuffin commented 1 year ago

TLDR: For optimal use on kubernetes, sidecars should not be messing with DNS at all.

PrivatePuffin commented 1 year ago

Any update on this? I'm trying to setup qbittorrent with mullvad wireguard using gluetun as sidecar but it just breaks access to qbittorrent webui whenever gluetun is started.

containers:
        - name: qbittorrent
          image: cr.hotio.dev/hotio/qbittorrent:latest
          env:
            - name: PUID
              value: "1000"
            - name: PGID
              value: "1000"
            - name: UMASK
              value: "002"
            - name: TZ
              value: "Europe/Berlin"
          ports:
            - name: qbittorrent-web
              containerPort: 8080
        - name: gluetun
          image: docker.io/qmcgaw/gluetun:latest
          securityContext:
            capabilities:
              add:
                - NET_ADMIN
          envFrom:
            - secretRef:
                name: qbittorrent-wireguard
          env:
            - name: TZ
              value: "Europe/Berlin"

EDIT: Needed to add:

- name: FIREWALL_INPUT_PORTS
              value: "8080"

to glueton container, now I have access to qbittorrent webui

Different issues, this issue is about DNS injection by gluetun

jhuynh08 commented 1 year ago

Hello,

I'm curious to see there is a workaround/fix for this. I'm having the exact same issue as @toniopelo. I have used DNS_KEEP_NAMESERVER switched to on and DOT switched off. I have also set the DNS_ADDRESS to my k8s DNS server to no availability.

I tested using the standard local domains for k8s in both containers with no luck. My current setup is going using nordvpn as the client and gluetun is setup as a sidecar.

@qdm12

PrivatePuffin commented 1 year ago

This is the same as #281 I believe. I'm working on a sort of UDP / DNS proxy to redirect DNS requests to Unbound (DNS over TLS) or the native DNS (from the Docker bridge or K8s) depending on the request. If it has no dot it's sent to the native DNS and otherwise sent to Unbound, such that Unbound block lists can't go through using the native Docker DNS.

Not sure that will solve it for K8s as this one might use dots in its private addresses, but I'll see what I can do. Doing a simple IP check on the result to check it's private could do the trick. Anyway, that'll take several days to finish, but I'll keep you updated. Thanks for your patience!

The correct solution for k8s, imho, is to simply offer the option not to have ANY nameserver being overwrithen.

We really want to move to GlueTun for our TrueCharts VPN addon, but this DNS overriding behavior is going to cause a boat load of unexpected behavior for our users.

graytonio commented 1 year ago

I was having a lot of trouble with this issue as well. After messing around with all the configs what finally did it was adding FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8. Not sure why because even when I tried to disable the firewall it did nothing but that config allowed the setup to work.

PrivatePuffin commented 1 year ago

I was having a lot of trouble with this issue as well. After messing around with all the configs what finally did it was adding FIREWALL_OUTBOUND_SUBNETS=10.0.0.0/8. Not sure why because even when I tried to disable the firewall it did nothing but that config allowed the setup to work.

That's not likely to be related though, the problem here isn't the firewall really, but the fact that a lot of applications only poll the primary/top DNS server and not the secondary.

graytonio commented 1 year ago

I understand that they shouldn't be related but changing that parameter was the difference between cluster dns working as intended and not working at all.

PrivatePuffin commented 1 year ago

I understand that they shouldn't be related but changing that parameter was the difference between cluster dns working as intended and not working at all.

What is your cluster cidr?