search

qdm12 / gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.
https://hub.docker.com/r/qmcgaw/gluetun
MIT License
7.55k stars 356 forks source link

Bug: Firewall rule issue on Dietpi affecting all Docker containers #171

Closed nicolasbuch closed 4 years ago

nicolasbuch commented 4 years ago

TLDR: Describe your issue in a one liner here

  1. Is this urgent?

    • [x] Yes
    • [ ] No

  2. What VPN service provider are you using?

    • [x] PIA
    • [ ] Mullvad
    • [ ] Windscribe
    • [ ] Surfshark

  3. What's the version of the program?

    Running version latest built on 2020-06-05T23:32:45Z (commit e33a6a8)

  4. What are you using to run the container?

    • [x] Docker run
    • [ ] Docker Compose
    • [ ] Kubernetes
    • [ ] Docker stack
    • [ ] Docker swarm
    • [ ] Podman
    • [ ] Other:

  5. Extra information

Logs:

=========================================
================ Gluetun ================
=========================================
==== A mix of OpenVPN, DNS over TLS, ====
======= Shadowsocks and Tinyproxy =======
========= all glued up with Go ==========
=========================================
=========== For tunneling to ============
======== your favorite VPN server =======
=========================================
=== Made with ❤️  by github.com/qdm12 ====
=========================================

Running version latest built on 2020-06-05T23:32:45Z (commit e33a6a8)

📣  New VPN provider supported surfshark.com

🔧  Need help? https://github.com/qdm12/private-internet-access-docker/issues/new
💻  Email? quentin.mcgaw@gmail.com
☕  Slack? Join from the Slack button on Github
💸  Help me? https://github.com/sponsors/qdm12
2020-06-08T10:34:12.440Z    INFO    OpenVPN version: 2.4.8
2020-06-08T10:34:12.464Z    INFO    Unbound version: 1.9.6
2020-06-08T10:34:12.482Z    INFO    IPtables version: v1.8.3
2020-06-08T10:34:12.537Z    INFO    TinyProxy version: 1.10.0
2020-06-08T10:34:12.555Z    INFO    ShadowSocks version: 3.3.4
2020-06-08T10:34:12.556Z    INFO    Settings summary below:
OpenVPN settings:
|--Network protocol: udp
|--Verbosity level: 1
|--Run as root: no
|--Target IP address: <nil>
|--Custom cipher:
|--Custom auth algorithm:
PIA settings:
 |--User: [redacted]
 |--Password: [redacted]
 |--Region: netherlands
 |--Encryption: strong
 |--Port forwarding: off
System settings:
|--User ID: 1000
|--Group ID: 1000
|--Timezone:
|--IP Status filepath: /ip
DNS over TLS settings:
 |--DNS over TLS provider:
  |--cloudflare
 |--Caching: enabled
 |--Block malicious: enabled
 |--Block surveillance: disabled
 |--Block ads: disabled
 |--Allowed hostnames:
  |--
 |--Private addresses:
  |--127.0.0.1/8
  |--10.0.0.0/8
  |--172.16.0.0/12
  |--192.168.0.0/16
  |--169.254.0.0/16
  |--::1/128
  |--fc00::/7
  |--fe80::/10
  |--::ffff:0:0/96
 |--Verbosity level: 1/5
 |--Verbosity details level: 0/4
 |--Validation log level: 0/2
 |--IPv6 resolution: disabled
 |--Update: every 24h0m0s
Firewall settings:
 |--Allowed subnets: 192.168.1.0/24, 192.168.100.0/24
TinyProxy settings: disabled
ShadowSocks settings: disabled

2020-06-08T10:34:12.559Z    INFO    openvpn configurator: checking for device /dev/net/tun
2020-06-08T10:34:12.559Z    WARN    TUN device is not available: open /dev/net/tun: no such file or directory
2020-06-08T10:34:12.559Z    INFO    openvpn configurator: creating /dev/net/tun
2020-06-08T10:34:12.561Z    INFO    openvpn configurator: writing auth file /etc/openvpn/auth.conf
2020-06-08T10:34:12.561Z    INFO    routing: detecting default network route
2020-06-08T10:34:12.562Z    INFO    routing: default route found: interface eth0, gateway 172.17.0.1, subnet 172.17.0.0/16
2020-06-08T10:34:12.562Z    INFO    firewall configurator: accepting all traffic
2020-06-08T10:34:12.709Z    INFO    Launching standard output merger
2020-06-08T10:34:12.710Z    INFO    routing: adding 192.168.1.0/24 as route via eth0
2020-06-08T10:34:12.717Z    INFO    routing: adding 192.168.100.0/24 as route via eth0
2020-06-08T10:34:12.720Z    INFO    firewall configurator: clearing all rules
2020-06-08T10:34:12.921Z    INFO    firewall configurator: blocking all traffic
2020-06-08T10:34:12.934Z    INFO    firewall configurator: creating general rules
2020-06-08T10:34:13.097Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.138.143 through eth0 on port udp 1197
2020-06-08T10:34:13.101Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.138.157 through eth0 on port udp 1197
2020-06-08T10:34:13.105Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.186.218 through eth0 on port udp 1197
2020-06-08T10:34:13.109Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.186.232 through eth0 on port udp 1197
2020-06-08T10:34:13.113Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.186.233 through eth0 on port udp 1197
2020-06-08T10:34:13.118Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.188.193 through eth0 on port udp 1197
2020-06-08T10:34:13.122Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.188.216 through eth0 on port udp 1197
2020-06-08T10:34:13.126Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.188.244 through eth0 on port udp 1197
2020-06-08T10:34:13.130Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.190.184 through eth0 on port udp 1197
2020-06-08T10:34:13.135Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.190.197 through eth0 on port udp 1197
2020-06-08T10:34:13.138Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.190.201 through eth0 on port udp 1197
2020-06-08T10:34:13.142Z    INFO    firewall configurator: allowing output traffic to VPN server 46.166.190.235 through eth0 on port udp 1197
2020-06-08T10:34:13.147Z    INFO    firewall configurator: allowing output traffic to VPN server 85.159.236.214 through eth0 on port udp 1197
2020-06-08T10:34:13.151Z    INFO    firewall configurator: allowing output traffic to VPN server 85.159.236.219 through eth0 on port udp 1197
2020-06-08T10:34:13.155Z    INFO    firewall configurator: allowing output traffic to VPN server 109.201.152.14 through eth0 on port udp 1197
2020-06-08T10:34:13.160Z    INFO    firewall configurator: allowing output traffic to VPN server 109.201.152.26 through eth0 on port udp 1197
2020-06-08T10:34:13.164Z    INFO    firewall configurator: allowing output traffic to VPN server 109.201.152.245 through eth0 on port udp 1197
2020-06-08T10:34:13.168Z    INFO    firewall configurator: allowing output traffic to VPN server 109.201.154.141 through eth0 on port udp 1197
2020-06-08T10:34:13.172Z    INFO    firewall configurator: allowing output traffic to VPN server 109.201.154.144 through eth0 on port udp 1197
2020-06-08T10:34:13.177Z    INFO    firewall configurator: allowing output traffic to VPN server 212.92.104.164 through eth0 on port udp 1197
2020-06-08T10:34:13.185Z    INFO    firewall configurator: accepting input and output traffic for 172.17.0.0/16
2020-06-08T10:34:13.192Z    INFO    firewall configurator: accepting input traffic through eth0 from 192.168.1.0/24 to 172.17.0.0/16
2020-06-08T10:34:13.196Z    INFO    firewall configurator: accepting output traffic through eth0 from 172.17.0.0/16 to 192.168.1.0/24
2020-06-08T10:34:13.199Z    INFO    firewall configurator: accepting input traffic through eth0 from 192.168.100.0/24 to 172.17.0.0/16
2020-06-08T10:34:13.204Z    INFO    firewall configurator: accepting output traffic through eth0 from 172.17.0.0/16 to 192.168.100.0/24
2020-06-08T10:34:13.208Z    INFO    openvpn: starting
2020-06-08T10:34:13.208Z    WARN    http server: restartOpenvpn function is not set, waiting...
2020-06-08T10:34:13.211Z    INFO    openvpn configurator: starting openvpn
2020-06-08T10:34:13.214Z    WARN    http server: restartUnbound function is not set, waiting...
2020-06-08T10:34:13.221Z    INFO    openvpn: OpenVPN 2.4.8 armv7-alpine-linux-musleabihf [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Feb  7 2020
2020-06-08T10:34:13.221Z    INFO    openvpn: library versions: OpenSSL 1.1.1g  21 Apr 2020, LZO 2.10
2020-06-08T10:34:13.232Z    INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]46.166.186.218:1197
2020-06-08T10:34:13.232Z    INFO    openvpn: UDP link local: (not bound)
2020-06-08T10:34:13.232Z    INFO    openvpn: UDP link remote: [AF_INET]46.166.186.218:1197
2020-06-08T10:34:13.232Z    INFO    openvpn: NOTE: UID/GID downgrade will be delayed because of --client, --pull, or --up-delay
2020-06-08T10:35:13.838Z    INFO    openvpn: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
2020-06-08T10:35:13.838Z    INFO    openvpn: TLS Error: TLS handshake failed
2020-06-08T10:35:13.838Z    INFO    openvpn: SIGUSR1[soft,tls-error] received, process restarting
2020-06-08T10:35:23.840Z    INFO    openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]46.166.188.244:1197
2020-06-08T10:35:23.841Z    INFO    openvpn: UDP link local: (not bound)
2020-06-08T10:35:23.841Z    INFO    openvpn: UDP link remote: [AF_INET]46.166.188.244:1197

Run Command(s) docker run -d --init --name=pia --restart always -e EXTRA_SUBNETS=192.168.1.0/24,192.168.100.0/24 -p 8112:8112 -p 58846:58846 -p 58946:58946 -p 30895:30895 -p 80:80 --cap-add=NET_ADMIN -e REGION="Netherlands" -e USER=***** -e PASSWORD=***** qmcgaw/private-internet-access

I have also tried minimal run command from doc docker run -d --name gluetun --cap-add=NET_ADMIN -e REGION="Netherlands" -e USER=**** -e PASSWORD=**** qmcgaw/private-internet-access

Host OS: Dietpi (Debian Buster 10.4)

Notes: Container does not seem to have connection to internet. All other containers also seem to loose internet connectivity when I run this container. In order to restore connectivity across my containers I have to stop and remove the PIA container and reboot host machine.

qdm12 commented 4 years ago

Hello!

nicolasbuch commented 4 years ago

Hello again

Sorry but this did not work for me. I don't know why i'm having this issue all of the sudden.

I have tried with the -e OPENVPN_TARGET_IP=46.166.190.201 but it made no difference. I have also pulled your latest changes on dockerhub (which I guess by now have the correct IP addresses) and it is still the same problem.

docker run -d --name gluetun --cap-add=NET_ADMIN -e REGION="Netherlands" -e USER=*** -e PASSWORD=**** qmcgaw/private-internet-access

All my containers run in network mode bridge per default.

I really hope that you might be able to help me out here, because i have no idea to what could be causing this. I have tried everything i can think of

nicolasbuch commented 4 years ago

If i do the exact same thing on my Raspberry Pi 4, I do not experience any of these issues. It just works.

The raspberry pi 4 and odroid XU4 are both ARM processors, both are running a fresh install of Linux DietPi 4.14.180 (buster) with the same configuration - and even the same docker version.

frepke commented 4 years ago

If i do the exact same thing on my Raspberry Pi 4, I do not experience any of these issues. It just works.

The raspberry pi 4 and odroid XU4 are both ARM processors, both are running a fresh install of Linux DietPi 4.14.180 (buster) with the same configuration - and even the same docker version.

Don't know if this is important, the RPi4 and Odroid XU4 are both ARM processors but with different architectures. The RPi4 has an Armv8-A architecture (64-bits), the Odroid has an Armv7-A architecture (32-bits).

qdm12 commented 4 years ago

Ok sorry for closing the issue too early!

That indeed feels like a firewall issue.

Maybe the docker daemon handles iptables differently from a cpu architecture to another (I doubt it though), thanks @Frepke for pointing this out.

What you can try is to run on your host:

iptables -nvL

before and after running the container, and comment back with the output. If a lot of rules change here, then it's indeed the gluetun firewall interfering with the host firewall. I'll do some google-fu if that's the case, or allow to disable the firewall with an env variable (as it was before, ironically).

Thanks

nicolasbuch commented 4 years ago

What you can try is to run on your host:

Before:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

After:

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    6   280 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    6   280 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    6   280 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    6   280 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    6   280 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    6   280 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
    6   280 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0
# Warning: iptables-legacy tables present, use iptables-legacy to see them

Running iptables-legacy -L outputs nothing:

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Updated: I have removed all networks to reduce cluttering of output

nicolasbuch commented 4 years ago

I don't see anything strange from the output above. I have cross checked it with the output from my raspberry pi 4. Firewall rules look the same both on the XU4 and on the PI4.

I have also tried compiling from source on the XU4 with same result as using the precompiled docker image.

frepke commented 4 years ago

Maybe you can give it a try with Armbian instead of DietPi on your Odroid. I had an HC1 myself running Armbian, never saw this issue on the Odroid.

met vriendelijke groet, Frepke

nicolasbuch commented 4 years ago

Maybe you can give it a try with Armbian instead of DietPi on your Odroid

Yea that will be my next step if we can't figure this out. I would just really like to have it working on Dietpi.

I had an HC1 myself running Armbian, never saw this issue on the Odroid

I have been running this setup for several years now on my XU4 (with Dietpi) and i never saw any issues like this either.

qdm12 commented 4 years ago

A few more questions:

  1. Is your host losing connection or just all your containers?
  2. Was gluetun ever working on your device?

Assuming 1. is just all your containers, try creating a docker network

docker network create testnet

and run gluetun using this network with --network testnet?

nicolasbuch commented 4 years ago

Is your host losing connection or just all your containers?

Just all the containers. The host always has internet.

Was gluetun ever working on your device?

I have been running with gluetun on my device for a long time. It has always worked like a charm. I decided to reinstall and now it doesn't work anymore.

Try creating a docker network

I've tried that already. I just tried it again with the same result. No internet in containers.

I will try to downgrade docker to an earlier version to see if it makes any difference.

qdm12 commented 4 years ago

I decided to reinstall

You mean your host os right?

How about if you restart the Docker daemon instead of rebooting?

Maybe try using an older commit to see if it works? See this how to in the wiki

nicolasbuch commented 4 years ago

I will try to downgrade docker to an earlier version to see if it makes any difference.

Downgraded docker to 18.06.1-ce, build e68fc7a with same result. So i think we can rule out that it happened due to docker being upgraded to most recent version.

You mean your host os right?

Yes exactly, sorry for not being clear enough

How about if you restart the Docker daemon instead of rebooting?

Restarting docker daemon does not restore internet connectivity to any running containers. Running new containers after daemon restart still don't have internet either. Only reboot seems to work here

Maybe try using an older commit to see if it works?

I did briefly try to docker build your v1 tag, even before i opened up this issue. I might give it another try with different commits to see if it makes any difference. Will report back my findings

qdm12 commented 4 years ago

I will add an undocumented environment variable to disable the firewall, for debugging purposes. At least we'll be able to find if it's a firewall issue or not. I'll comment back once it's done.

qdm12 commented 4 years ago

I've added FIREWALL you can try to set to -e FIREWALL=off and it won't touch iptables, let's see if it works.

nicolasbuch commented 4 years ago

I've added FIREWALL you can try to set to -e FIREWALL=off and it won't touch iptables, let's see if it works.

Amazing, this actually worked! I now have internet in the container. So it is the firewall rules are making the trouble.

qdm12 commented 4 years ago

Alright cool!

What we should try is find what's the rule messing up your network. Let's try the first command which blocks everything... which should only block it in the container.

docker run -it --rm --cap-add=NET_ADMIN alpine:3.11
apk add iptables
# clear rules
iptables --flush
iptables --delete-chain
iptables -t nat --flush
iptables -t nat --delete-chain
# that should work:
ping -c 1 google.com 
# block everything
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -P FORWARD DROP
# check if other containers lost connection, they shouldn't
# if they did lost connection, re-accept all traffic
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
# that should work
ping -c 1 google.com 
# verify if the containers connection is up now?
nicolasbuch commented 4 years ago

Amazing, it seems to the clearing of iptables rules that is messing with my network.

After i run following commands, the container looses internet connectivity: 

docker run -it --rm --cap-add=NET_ADMIN alpine:3.11
apk add iptables
# clear rules
iptables --flush
iptables --delete-chain
iptables -t nat --flush
iptables -t nat --delete-chain

I went through them one by one to check at which command it actually looses internet and it seems to be after i run:

iptables -t nat --flush
qdm12 commented 4 years ago

Ok, and if you omit this iptables -t nat --flush command do you lose connectivity? If that's the only rule causing issues, I'll check if it's actually necessary as it might not be.

orstendium commented 4 years ago

I've been facing the same issue, also on an odroid xu4 but I'm using Armbian.

For me, both iptables -t nat --flush and iptables -t nat --delete-chain need to be omitted or else I lose connectivity.

qdm12 commented 4 years ago

Thanks @CDNHammer for sharing.

As Docker uses NAT (network address translation) to 'port forward' ports of containers, I guess, on some systems, changing the nat table messes up with the host firewall/ docker networking. There is not much I could find online about that.

Anyway, these nat flushing rules shouldn't be necessary, I'll remove them. Thanks!

qdm12 commented 4 years ago

I removed the nat flushing rules, can you please try with the standard (:latest) docker image? (also wait for the build to finish)

nicolasbuch commented 4 years ago

I've been facing the same issue, also on an odroid xu4 but I'm using Armbian

Yes, thanks for sharing @CDNHammer. Great to know i'm not the only one facing these issues.

For me, both iptables -t nat --flush and iptables -t nat --delete-chain need to be omitted or else I lose connectivity.

The same goes for me. Both are causing issues.

I removed the nat flushing rules, can you please try with the standard (:latest) docker image?

@qdm12 finally it works, thank you! Now it just leaves me wondering. Why are we not able to flush nat? What could be causing this. To my knowledge it is neither Docker nor DietPi. Could this be a bug in the linux kernel on the XU4?

qdm12 commented 4 years ago

I'm curious too but don't want to spend my next days digging on that :smile: It's likely Docker though (the daemon) which doesn't handle network isolation correctly on that particular os/kernel/cpu arch. Especially since your host still has connection but all the containers lose it. Anyway, let's close this issue. If you ever find why I would be curious :wink: !