Bug: Significant bandwidth reduction only when connected via Gluetun

NikolausC commented 1 year ago

Is this urgent?

None

Host OS

Debian 11 (bullseye)

CPU arch

aarch64

VPN service provider

Mullvad

What are you using to run the container

Podman

What is the version of Gluetun

Running version latest built on 2022-12-15T10:02:42.178Z (commit 490693b)

What's the problem 🤔

When making network requests via Gluetun-connected containers, I always see a significant reduction in bandwidth. If the system is connected to the VPN, download speeds are 500Mbps+. When connected via Gluetun, speeds are around 60-85Mbps.

I've read the wiki page about bandwidth speeds
- I've tried several nearby countries
- I've tried OpenVPN and Wireguard configs
- I'm running Podman on Linux, so no virtualisation going on
- When running speedtest-cli in a container, the reduction in bandwidth is only seen if the container is connected via Gluetun. If instead the OS is connected to the VPN, then running speedtest-cli gives results pretty close to full speed. This should rule out any potential throttling that Mullvad might be doing, as that would also be seen without using Gluetun.
I've also carried out speed tests in the Gluetun containers directly, rather than having another container share its network stack with '--network=container:...' to rule out potential issues caused by that.

There are four sections in the config I've shared below:

Speed test container using native internet connection
Speed tests for a Gluetun container connected via Wireguard
Speed tests for a Gluetun container connected via OpenVPN
Speed test container with Wireguard VPN connection

I've found this thread on Reddit with someone using Gluetun, OpenVPN, and Private Internet Access who appears to be seeing the same thing.

I'm running containers in Podman 4.3.1 (built from source) running on a Raspberry Pi 4B.

If there's any more information I can provide, please do let me know.

Share your logs

2022-12-23T13:10:27Z INFO [routing] default route found: interface tap0, gateway 10.0.2.2 and assigned IP 10.0.2.100
2022-12-23T13:10:27Z INFO [routing] default route found: interface tap0, gateway fe80::2 and assigned IP fd00::f452:a3ff:fed7:8ef6
2022-12-23T13:10:27Z INFO [routing] local ethernet link found: tap0
2022-12-23T13:10:27Z INFO [routing] local ipnet found: 10.0.2.0/24
2022-12-23T13:10:27Z INFO [routing] local ipnet found: fd00::/64
2022-12-23T13:10:27Z INFO [routing] local ipnet found: fe80::/64
2022-12-23T13:10:27Z INFO [firewall] enabling...
2022-12-23T13:10:28Z INFO [firewall] enabled successfully
2022-12-23T13:10:28Z INFO [storage] creating /gluetun/servers.json with 13224 hardcoded servers
2022-12-23T13:10:29Z INFO Alpine version: 3.16.3
2022-12-23T13:10:29Z INFO OpenVPN 2.4 version: 2.4.12
2022-12-23T13:10:29Z INFO OpenVPN 2.5 version: 2.5.6
2022-12-23T13:10:29Z INFO Unbound version: 1.15.0
2022-12-23T13:10:29Z INFO IPtables version: v1.8.8
2022-12-23T13:10:29Z INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: mullvad
|   |   └── Server selection settings:
|   |       ├── VPN type: wireguard
|   |       ├── Countries: sweden
|   |       └── Wireguard selection settings:
|   └── Wireguard settings:
|       ├── Private key: mK...04=
|       ├── Interface addresses:
|       |   └── 10.64.68.213/32
|       └── Network interface: tun0
├── DNS settings:
|   ├── DNS server address to use: 127.0.0.1
|   ├── Keep existing nameserver(s): no
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: yes
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:7f00:1/104
|               ├── ::ffff:a00:0/104
|               ├── ::ffff:a9fe:0/112
|               ├── ::ffff:ac10:0/108
|               └── ::ffff:c0a8:0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: INFO
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 1000
|   └── Process GID: 1000
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   └── IP file path: /tmp/gluetun/ip
└── Version settings:
    └── Enabled: yes
2022-12-23T13:10:29Z INFO [routing] default route found: interface tap0, gateway 10.0.2.2 and assigned IP 10.0.2.100
2022-12-23T13:10:29Z INFO [routing] default route found: interface tap0, gateway fe80::2 and assigned IP fd00::f452:a3ff:fed7:8ef6
2022-12-23T13:10:29Z INFO [routing] adding route for 0.0.0.0/0
2022-12-23T13:10:29Z INFO [routing] adding route for ::/0
2022-12-23T13:10:29Z INFO [firewall] setting allowed subnets...
2022-12-23T13:10:29Z INFO [routing] default route found: interface tap0, gateway 10.0.2.2 and assigned IP 10.0.2.100
2022-12-23T13:10:29Z INFO [routing] default route found: interface tap0, gateway fe80::2 and assigned IP fd00::f452:a3ff:fed7:8ef6
2022-12-23T13:10:29Z INFO [dns over tls] using plaintext DNS at address 1.1.1.1
2022-12-23T13:10:29Z INFO [http server] http server listening on [::]:8000
2022-12-23T13:10:29Z INFO [healthcheck] listening on 127.0.0.1:9999
2022-12-23T13:10:29Z INFO [firewall] allowing VPN connection...
2022-12-23T13:10:29Z INFO [wireguard] Using available kernelspace implementation
2022-12-23T13:10:29Z INFO [wireguard] Connecting to [2a03:1b20:4:f011::a09f]:51820
2022-12-23T13:10:29Z INFO [wireguard] Wireguard is up
2022-12-23T13:10:29Z INFO [dns over tls] downloading DNS over TLS cryptographic files
2022-12-23T13:10:37Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2022-12-23T13:10:37Z INFO [vpn] stopping
2022-12-23T13:10:37Z ERROR [vpn] cannot get version information: Get "https://api.github.com/repos/qdm12/gluetun/commits": context canceled
2022-12-23T13:10:37Z INFO [vpn] starting
2022-12-23T13:10:37Z INFO [firewall] allowing VPN connection...
2022-12-23T13:10:37Z INFO [wireguard] Using available kernelspace implementation
2022-12-23T13:10:37Z INFO [wireguard] Connecting to 185.195.233.70:51820
2022-12-23T13:10:37Z INFO [wireguard] Wireguard is up
2022-12-23T13:10:37Z INFO [healthcheck] healthy!
2022-12-23T13:10:39Z WARN [dns over tls] cannot update files: Get "https://www.internic.net/domain/named.root": dial tcp: lookup www.internic.net on 1.1.1.1:53: read udp 10.64.68.213:41054->1.1.1.1:53: i/o timeout
2022-12-23T13:10:39Z INFO [dns over tls] attempting restart in 10s
2022-12-23T13:10:49Z INFO [dns over tls] downloading DNS over TLS cryptographic files
2022-12-23T13:10:50Z INFO [dns over tls] downloading hostnames and IP block lists
2022-12-23T13:11:00Z INFO [healthcheck] unhealthy: cannot dial: dial tcp4: lookup cloudflare.com: i/o timeout(see https://github.com/qdm12/gluetun/wiki/Healthcheck)
2022-12-23T13:11:02Z INFO [dns over tls] init module 0: validator
2022-12-23T13:11:02Z INFO [dns over tls] init module 1: iterator
2022-12-23T13:11:02Z INFO [dns over tls] start of service (unbound 1.15.0).
2022-12-23T13:11:02Z INFO [dns over tls] generate keytag query _ta-4a5c-4f66. NULL IN
2022-12-23T13:11:02Z INFO [dns over tls] generate keytag query _ta-4a5c-4f66. NULL IN
2022-12-23T13:11:03Z INFO [healthcheck] healthy!
2022-12-23T13:11:03Z INFO [dns over tls] ready

Share your configuration

# (1) Native download speed test (No VPN)
wget -q -O- https://am.i.mullvad.net/connected
# You are not connected to Mullvad. Your IP address is REDACTED

podman run --rm gists/speedtest-cli
# Server: razorblue - Leeds (id = 17449)
# ISP: Virgin Media
# Latency:    19.90 ms   (1.93 ms jitter)
# Download:   552.33 Mbps (data used: 720.5 MB )
# Upload:    37.21 Mbps (data used: 28.9 MB )
# Packet Loss:     0.0%

# (2) Testing Gluetun (Wireguard, Mullvad)
podman run --name wireguard_vpn \
    --detach \
    --cap-add=NET_ADMIN \
    -e VPN_SERVICE_PROVIDER=mullvad \
    -e VPN_TYPE=wireguard \
    -e WIREGUARD_PRIVATE_KEY=REDACTED \
    -e WIREGUARD_ADDRESSES="10.64.68.213/32" \
    -e SERVER_COUNTRIES=sweden \
    --device /dev/net/tun \
    qmcgaw/gluetun

podman run --rm --network=container:wireguard_vpn alpine /bin/sh -c 'wget -q -O- https://am.i.mullvad.net/connected'
# You are connected to Mullvad (server se18-wireguard). Your IP address is 193.138.218.219

podman run --rm --network=container:wireguard_vpn gists/speedtest-cli
# Server: Bahnhof AB - Malmo (id = 34338)
# ISP: 31173 Services AB
# Latency:    36.49 ms   (2.15 ms jitter)
# Download:    65.70 Mbps (data used: 91.5 MB )
# Upload:    35.19 Mbps (data used: 38.3 MB )
# Packet Loss:     0.0%

podman exec -it wireguard_vpn /bin/sh -c 'apk update && apk add speedtest-cli && speedtest-cli'
# Download: 68.46 Mbit/s
# Upload: 33.16 Mbit/s

podman rm -f wireguard_vpn

# (3) Testing Gluetun (OpenVPN, Mullvad)
podman run --name open_vpn \
    --detach \
    --cap-add=NET_ADMIN \
    -e VPN_SERVICE_PROVIDER=mullvad \
    -e VPN_TYPE=openvpn \
    -e OPENVPN_USER=REDACTED \
    -e SERVER_COUNTRIES=sweden \
    --device /dev/net/tun:/dev/net/tun \
    qmcgaw/gluetun

podman run --rm --network=container:open_vpn alpine /bin/sh -c 'wget -q -O- https://am.i.mullvad.net/connected'
# You are connected to Mullvad (server se-got-011). Your IP address is 185.213.154.171

podman run --rm --network=container:open_vpn gists/speedtest-cli
# Server: Telenor AB - Göteborg (id = 35925)
# ISP: 31173 Services AB
# Latency:    40.17 ms   (1.78 ms jitter)
# Download:    82.88 Mbps (data used: 106.2 MB )
# Upload:    34.50 Mbps (data used: 34.8 MB )
# Packet Loss:    16.8%

podman exec -it open_vpn /bin/sh -c 'apk update && apk add speedtest-cli && speedtest-cli'
# Download: 29.35 Mbit/s
# Upload: 31.96 Mbit/s

podman rm -f open_vpn

# (4) Connect system to VPN
wg-quick up se28-wireguard

wget -q -O- https://am.i.mullvad.net/connected
# You are connected to Mullvad (server se28-wireguard). Your IP address is 185.195.233.204

podman run --rm gists/speedtest-cli
# Server: A3 Allmänna IT - och Telekomaktiebolaget - Stockholm (id = 20783)
# ISP: 31173 Services AB
# Latency:    48.11 ms   (3.75 ms jitter)
# Download:   524.13 Mbps (data used: 695.6 MB )
# Upload:    35.31 Mbps (data used: 38.1 MB )
# Packet Loss:     0.0%

qdm12 commented 1 year ago

Yeah there is a bottleneck somewhere in docker/podman/lxc when using a vpn (both openvpn and wireguard). Using other vpn images produce the same bottleneck. If you find the root cause/solution for this, let me know though. Let's keep this opened if someone stumbles on the same problem and finds a solution.

pwn2w1n commented 1 year ago

I have the same issue also using mullvad. Worked fine till about a week or two ago where my speeds dropped from around 2.5-3.5MB/S to under 10KB/S only on gluetun, speeds are fine when using the app on my PC or phone.

edit: nevermind, in my case it seems the problem is qbittorent. Set up a transmission docker container with gluetun for the network mode and it works fine. Apologies.

Hackermanswitch commented 1 year ago

same issue here; using ovpn from netflixvpn and tried as well with wind scribe.

qdm12 / gluetun