Bug: Really slow speeds (wireguard and openvpn, PIA and windscribe)

vic1707 commented 1 week ago

Is this urgent?

No

Host OS

Almalinux 9.4 VM running on a Proxmox server, the VM has 8 cores of a EPYC 7551P 32cores

CPU arch

x86_64

VPN service provider

Windscribe

What are you using to run the container

Podman

What is the version of Gluetun

3.39.1

What's the problem 🤔

I'm getting really slow speeds but have no idea why nor how to properly debug everything, find bottlenecks etc 😓.

Before switching to gluetun I was running haugene's transmission + openvpn combo (using PIA as a provider), I was getting speeds pretty close to a full 1Gbps.

Now that I moved to gluetun I'm happy when I get over 70Mbps.

I tried gluetun (as the only running container) in wireguard windscribe mode (as described by the compose file) but also tried PIA in openvpn mode in case wg-easy was messing with wireguard mode. All after a fresh podman system reset, and trying minimal configs (no cap_add/cap_drop /security_opt/networks/sysctls/different servers, regions/not being inside a pod etc... I've never seen any configuration go past the 120Mbps.

Local speeds:

[marina@marina containers]$ podman run --rm \
    tianon/speedtest speedtest --accept-license  --accept-gdpr --server-id 21606
==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

        https://www.speedtest.net/about/eula
        https://www.speedtest.net/about/terms
        https://www.speedtest.net/about/privacy

==============================================================================

License acceptance recorded. Continuing.

==============================================================================

Ookla collects certain data through Speedtest that may be considered
personally identifiable, such as your IP address, unique device
identifiers or location. Ookla believes it has a legitimate interest
to share this data with internet providers, hardware manufacturers and
industry regulators to help them understand and create a better and
faster internet. For further information including how the data may be
shared, where the data may be transferred and Ookla's contact details,
please see our Privacy Policy at:

       http://www.speedtest.net/privacy

==============================================================================

License acceptance recorded. Continuing.

   Speedtest by Ookla

      Server: S&A Telephone - Allen, KS (id: 21606)
         ISP: Orange
Idle Latency:   140.98 ms   (jitter: 0.23ms, low: 140.81ms, high: 141.19ms)
    Download:   700.27 Mbps (data used: 1.1 GB)
                141.33 ms   (jitter: 0.89ms, low: 140.24ms, high: 153.19ms)
      Upload:   235.85 Mbps (data used: 410.3 MB)
                142.56 ms   (jitter: 6.46ms, low: 140.10ms, high: 355.65ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/217aadf3-d22c-486a-8e0a-9ce86297bb6b

Via gluetun

[marina@marina containers]$ podman run --rm \
    --pod=pod_containers --network=container:gluetun \
    tianon/speedtest speedtest --accept-license  --accept-gdpr --server-id 21606

==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

        https://www.speedtest.net/about/eula
        https://www.speedtest.net/about/terms
        https://www.speedtest.net/about/privacy

==============================================================================

License acceptance recorded. Continuing.

   Speedtest by Ookla

      Server: S&A Telephone - Allen, KS (id: 21606)
         ISP: GSL Networks
Idle Latency:   397.10 ms   (jitter: 1.08ms, low: 396.89ms, high: 398.90ms)
    Download:     6.81 Mbps (data used: 10.5 MB)
                626.19 ms   (jitter: 88.44ms, low: 395.71ms, high: 1781.11ms)
      Upload:    69.06 Mbps (data used: 114.1 MB)
                946.05 ms   (jitter: 91.22ms, low: 397.68ms, high: 2962.71ms)
 Packet Loss:     2.7%
  Result URL: https://www.speedtest.net/result/c/86e5c85e-5799-44cd-af0b-1eedd80739eb

And a second run cause the first one seemd slow

[marina@marina containers]$ podman run --rm     --pod=pod_containers --network=container:gluetun     tianon/speedtest speedtest --accept-license  --accept-gdpr --server-id 21606
==============================================================================

You may only use this Speedtest software and information generated
from it for personal, non-commercial use, through a command line
interface on a personal computer. Your use of this software is subject
to the End User License Agreement, Terms of Use and Privacy Policy at
these URLs:

        https://www.speedtest.net/about/eula
        https://www.speedtest.net/about/terms
        https://www.speedtest.net/about/privacy

==============================================================================

License acceptance recorded. Continuing.

   Speedtest by Ookla

      Server: S&A Telephone - Allen, KS (id: 21606)
         ISP: GSL Networks
Idle Latency:   396.73 ms   (jitter: 0.29ms, low: 396.60ms, high: 397.30ms)
    Download:    16.08 Mbps (data used: 23.4 MB)
                650.31 ms   (jitter: 94.64ms, low: 393.62ms, high: 2256.43ms)
      Upload:    33.17 Mbps (data used: 51.1 MB)
                675.65 ms   (jitter: 89.28ms, low: 399.35ms, high: 2331.46ms)
 Packet Loss:     2.3%
  Result URL: https://www.speedtest.net/result/c/9bc7a3b9-5d1b-4c8b-b02b-27c7ccd7397e

As an added note, my wg-easy is also just as slow, but given my upload speeds I'm not that surprised.

For now my best guesses are:

My VM is broken in some sense despite never doing anything else than running containers (will try to format and reinstall it when I find time).
The linux kernel used by Almalinux is too old, v5 vs v6 for the latest major release (will also try to move to Fedora or other whenever possible).

But I would love to get some help to figure out what's going on

Share your logs (at least 10 lines)

========================================
========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version v3.39.1 built on 2024-09-29T18:16:23.495Z (commit 67ae5f5)

📣 All control server routes will become private by default after the v3.41.0 release

🔧 Need help? ☕ Discussion? https://github.com/qdm12/gluetun/discussions/new/choose
🐛 Bug? ✨ New feature? https://github.com/qdm12/gluetun/issues/new/choose
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2024-11-13T22:57:14+01:00 INFO [routing] default route found: interface eth0, gateway 10.99.0.1, assigned IP 10.99.0.62 and family v4
2024-11-13T22:57:14+01:00 INFO [routing] local ethernet link found: eth0
2024-11-13T22:57:14+01:00 INFO [routing] local ipnet found: 10.99.0.0/26
2024-11-13T22:57:14+01:00 INFO [firewall] enabling...
2024-11-13T22:57:14+01:00 INFO [firewall] enabled successfully
2024-11-13T22:57:14+01:00 INFO [storage] creating /gluetun/servers.json with 20478 hardcoded servers
2024-11-13T22:57:15+01:00 INFO Alpine version: 3.20.3
2024-11-13T22:57:15+01:00 INFO OpenVPN 2.5 version: 2.5.10
2024-11-13T22:57:15+01:00 INFO OpenVPN 2.6 version: 2.6.11
2024-11-13T22:57:15+01:00 INFO Unbound version: 1.20.0
2024-11-13T22:57:15+01:00 INFO IPtables version: v1.8.10
2024-11-13T22:57:15+01:00 INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: windscribe
|   |   └── Server selection settings:
|   |       ├── VPN type: wireguard
|   |       └── Wireguard selection settings:
|   └── Wireguard settings:
|       ├── Private key: COz...mE=
|       ├── Pre-shared key: WF9...f4=
|       ├── Interface addresses:
|       |   └── 100.108.130.154/32
|       ├── Allowed IPs:
|       |   ├── 0.0.0.0/0
|       |   └── ::/0
|       └── Network interface: tun0
|           └── MTU: 1400
├── DNS settings:
|   ├── Keep existing nameserver(s): no
|   ├── DNS server address to use: 127.0.0.1
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: yes
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:127.0.0.1/104
|               ├── ::ffff:10.0.0.0/104
|               ├── ::ffff:169.254.0.0/112
|               ├── ::ffff:172.16.0.0/108
|               └── ::ffff:192.168.0.0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: info
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Duration to wait after success: 5s
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   ├── Logging: yes
|   └── Authentication file path: /gluetun/auth/config.toml
├── OS Alpine settings:
|   ├── Process UID: 1000
|   ├── Process GID: 1000
|   └── Timezone: Europe/Paris
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   ├── IP file path: /tmp/gluetun/ip
|   └── Public IP data API: ipinfo
└── Version settings:
    └── Enabled: yes
2024-11-13T22:57:15+01:00 INFO [routing] default route found: interface eth0, gateway 10.99.0.1, assigned IP 10.99.0.62 and family v4
2024-11-13T22:57:15+01:00 INFO [routing] adding route for 0.0.0.0/0
2024-11-13T22:57:15+01:00 INFO [firewall] setting allowed subnets...
2024-11-13T22:57:15+01:00 INFO [routing] default route found: interface eth0, gateway 10.99.0.1, assigned IP 10.99.0.62 and family v4
2024-11-13T22:57:15+01:00 INFO [dns] using plaintext DNS at address 1.1.1.1
2024-11-13T22:57:15+01:00 INFO [http server] http server listening on [::]:8000
2024-11-13T22:57:15+01:00 INFO [healthcheck] listening on 127.0.0.1:9999
2024-11-13T22:57:15+01:00 INFO [firewall] allowing VPN connection...
2024-11-13T22:57:15+01:00 INFO [wireguard] Using available kernelspace implementation
2024-11-13T22:57:15+01:00 INFO [wireguard] Connecting to 103.107.198.228:1194
2024-11-13T22:57:15+01:00 INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-11-13T22:57:15+01:00 INFO [dns] downloading DNS over TLS cryptographic files
2024-11-13T22:57:16+01:00 INFO [healthcheck] healthy!
2024-11-13T22:57:21+01:00 INFO [dns] downloading hostnames and IP block lists
2024-11-13T22:57:31+01:00 INFO [dns] init module 0: validator
2024-11-13T22:57:31+01:00 INFO [dns] init module 1: iterator
2024-11-13T22:57:31+01:00 INFO [dns] start of service (unbound 1.20.0).
2024-11-13T22:57:32+01:00 INFO [dns] generate keytag query _ta-4a5c-4f66-9728. NULL IN
2024-11-13T22:57:32+01:00 INFO [dns] generate keytag query _ta-4a5c-4f66-9728. NULL IN
2024-11-13T22:57:33+01:00 INFO [dns] ready
2024-11-13T22:57:35+01:00 INFO [ip getter] Public IP address is 103.107.198.231 (Singapore, Singapore, Singapore)
2024-11-13T22:57:36+01:00 INFO [vpn] You are running the latest release v3.39.1

Share your configuration

# somewhat truncated version to only include gluetun
# complete setup can be found here https://github.com/vic1707/homelab-config/blob/main/hydra/marina/containers/compose.yml (along with the setup script for the Almalinux VM

# other services running:
# authelia
# caddy
# gickup
# jellyfin/jellyfin
# transmission (custom image with only transmission, not haugene nor linuxserver images)
# wg-easy

networks:
    shared:
        driver: bridge
        ipam:
            config:
                - subnet: 10.99.0.0/26
        name: shared
secrets:
    wireguard_addresses:
        external: true
    wireguard_preshared_key:
        external: true
    wireguard_private_key:
        external: true
services:
    gluetun:
        cap_drop: # actually not used because of the privileged: true (expected)
            - ALL
        container_name: gluetun
        environment:
            TZ: Europe/Paris
            VPN_SERVICE_PROVIDER: windscribe
            VPN_TYPE: wireguard
        image: docker.io/qmcgaw/gluetun:v3.39.1
        networks:
            shared:
                ipv4_address: 10.99.0.62
        privileged: true
        read_only: false
        restart: always
        secrets:
            - wireguard_private_key
            - wireguard_addresses
            - wireguard_preshared_key
        security_opt:
            - no-new-privileges=true
        sysctls:
            net.ipv6.conf.all.disable_ipv6: 1

github-actions[bot] commented 1 week ago

@qdm12 is more or less the only maintainer of this project and works on it in his free time. Please:

do not ask for updates, be patient
:+1: the issue to show your support instead of commenting @qdm12 usually checks issues at least once a week, if this is a new urgent bug, revert to an older tagged container image

qdm12 commented 1 week ago

Before switching to gluetun I was running haugene's transmission + openvpn combo (using PIA as a provider), I was getting speeds pretty close to a full 1Gbps.

In Docker in your VM? If not, you have your answer: docker and/or the VM are the bottlenecks.

See https://github.com/qdm12/gluetun-wiki/blob/main/faq/bandwidth.md

One more thing not documented (since I'm working on #2586 so there won't need to document it anymore) is try to increase WIREGUARD_MTU from 1400 to a bit higher. It might become unstable and give you TLS errors notably though, and might not help at all either, but worth a try.

in case wg-easy was messing with wireguard mode.

There is no wg-easy in Gluetun, it's pretty much custom Go code I wrote either interacting with the kernel (kernelspace) or wireguard-go (userspace).

Closing this since this is not a bug, and I cannot do anything really except pointing you in the right direction. FYI I run gluetun with docker on an arch linux host and get near gigabit bandwidth.

github-actions[bot] commented 1 week ago

Closed issues are NOT monitored, so commenting here is likely to be not seen. If you think this is still unresolved and have more information to bring, please create another issue.

This is an automated comment setup because @qdm12 is the sole maintainer of this project which became too popular to monitor issues closed.

vic1707 commented 1 week ago

Hi, thx for the reply @qdm12

In Docker in your VM? If not, you have your answer: docker and/or the VM are the bottlenecks.

In podman in the VM, the same one as I'm using for gluetun I had a stack with haugene transmission-openvpn and replaced it with gluetun, everything else is unchanged.

try to increase WIREGUARD_MTU from 1400 to a bit higher.

thx I'll try that !

There is no wg-easy in Gluetun, it's pretty much custom Go code I wrote either interacting with the kernel (kernelspace) or wireguard-go (userspace).

Yup I know, sorry I probably wasn't clear enough, what I meant is that alonside gluetun I have a wg-easy container running, and I thought it could be messing with gluetun (2 containers accessing the same module or sometihng like that). But the behavior is the same even when its turned off so likely not the issue.

Closing this since this is not a bug, and I cannot do anything really except pointing you in the right direction. FYI I run gluetun with docker on an arch linux host and get near gigabit bandwidth.

Okay thx for the info ! I'll continue to look for a fix on my side.

qdm12 / gluetun