search

qdm12 / gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.
https://hub.docker.com/r/qmcgaw/gluetun
MIT License
8.04k stars 371 forks source link

Bug: Memory overflow on linux alpine #1687

Closed Lumino2 closed 1 year ago

Lumino2 commented 1 year ago

Is this urgent?

No

Host OS

Alpine

CPU arch

x86_64

VPN service provider

Surfshark

What are you using to run the container

docker-compose

What is the version of Gluetun

Running version latest built on 2023-06-12T13:56:16.720Z (commit 83826e1)

What's the problem 🤔

I'm running proxmox => linux alpine => docker (gluetun + qbittorrent) The setup works correctly but as soon as traffic is sent through gluetun RAM usage builds up rapidly until the server becomes really slow. When gluetun is idle, memory usage stays the same level. I have 16GB of ram dedictated to the vm that only runs 2 docker instances which should be plenty. I don't see the cause in the logs. Anyone can help? Capture

Share your logs

docker container logs gluetun
========================================
========================================
=============== gluetun ================
========================================
=========== Made with ❤️ by ============
======= https://github.com/qdm12 =======
========================================
========================================

Running version latest built on 2023-06-12T13:56:16.720Z (commit 83826e1)

🔧 Need help? https://github.com/qdm12/gluetun/discussions/new
🐛 Bug? https://github.com/qdm12/gluetun/issues/new
✨ New feature? https://github.com/qdm12/gluetun/issues/new
☕ Discussion? https://github.com/qdm12/gluetun/discussions/new
💻 Email? quentin.mcgaw@gmail.com
💰 Help me? https://www.paypal.me/qmcgaw https://github.com/sponsors/qdm12
2023-06-14T09:55:43+01:00 INFO [routing] default route found: interface eth0, gateway 172.18.0.1, assigned IP 172.18.0.3 and family v4
2023-06-14T09:55:43+01:00 INFO [routing] local ethernet link found: eth0
2023-06-14T09:55:43+01:00 INFO [routing] local ipnet found: 172.18.0.0/16
2023-06-14T09:55:43+01:00 INFO [firewall] enabling...
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --policy INPUT DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --policy OUTPUT DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --policy FORWARD DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --policy INPUT DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --policy OUTPUT DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --policy FORWARD DROP
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append INPUT -i lo -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --append INPUT -i lo -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append OUTPUT -o lo -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --append OUTPUT -o lo -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --append OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --append INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append OUTPUT -o eth0 -s 172.18.0.3 -d 172.18.0.0/16 -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] ip6tables-nft --append OUTPUT -o eth0 -d ff02::1:ff/104 -j ACCEPT
2023-06-14T09:55:43+01:00 DEBUG [firewall] iptables --append INPUT -i eth0 -d 172.18.0.0/16 -j ACCEPT
2023-06-14T09:55:43+01:00 INFO [firewall] enabled successfully
2023-06-14T09:55:44+01:00 INFO [storage] merging by most recent 17678 hardcoded servers and 17678 servers read from /gluetun/servers.json
2023-06-14T09:55:44+01:00 DEBUG [netlink] IPv6 is not supported after searching 0 routes
2023-06-14T09:55:44+01:00 INFO Alpine version: 3.18.0
2023-06-14T09:55:44+01:00 INFO OpenVPN 2.5 version: 2.5.8
2023-06-14T09:55:44+01:00 INFO OpenVPN 2.6 version: 2.6.4
2023-06-14T09:55:44+01:00 INFO Unbound version: 1.17.1
2023-06-14T09:55:44+01:00 INFO IPtables version: v1.8.9
2023-06-14T09:55:44+01:00 INFO Settings summary:
├── VPN settings:
|   ├── VPN provider settings:
|   |   ├── Name: surfshark
|   |   └── Server selection settings:
|   |       ├── VPN type: openvpn
|   |       ├── Countries: netherlands
|   |       └── OpenVPN server selection settings:
|   |           └── Protocol: UDP
|   └── OpenVPN settings:
|       ├── OpenVPN version: 2.5
|       ├── User: [set]
|       ├── Password: [set]
|       ├── Network interface: tun0
|       ├── Run OpenVPN as: root
|       └── Verbosity level: 1
├── DNS settings:
|   ├── DNS server address to use: 127.0.0.1
|   ├── Keep existing nameserver(s): no
|   └── DNS over TLS settings:
|       ├── Enabled: yes
|       ├── Update period: every 24h0m0s
|       ├── Unbound settings:
|       |   ├── Authoritative servers:
|       |   |   └── cloudflare
|       |   ├── Caching: yes
|       |   ├── IPv6: no
|       |   ├── Verbosity level: 1
|       |   ├── Verbosity details level: 0
|       |   ├── Validation log level: 0
|       |   ├── System user: root
|       |   └── Allowed networks:
|       |       ├── 0.0.0.0/0
|       |       └── ::/0
|       └── DNS filtering settings:
|           ├── Block malicious: yes
|           ├── Block ads: no
|           ├── Block surveillance: no
|           └── Blocked IP networks:
|               ├── 127.0.0.1/8
|               ├── 10.0.0.0/8
|               ├── 172.16.0.0/12
|               ├── 192.168.0.0/16
|               ├── 169.254.0.0/16
|               ├── ::1/128
|               ├── fc00::/7
|               ├── fe80::/10
|               ├── ::ffff:127.0.0.1/104
|               ├── ::ffff:10.0.0.0/104
|               ├── ::ffff:169.254.0.0/112
|               ├── ::ffff:172.16.0.0/108
|               └── ::ffff:192.168.0.0/112
├── Firewall settings:
|   └── Enabled: yes
├── Log settings:
|   └── Log level: DEBUG
├── Health settings:
|   ├── Server listening address: 127.0.0.1:9999
|   ├── Target address: cloudflare.com:443
|   ├── Duration to wait after success: 5s
|   ├── Read header timeout: 100ms
|   ├── Read timeout: 500ms
|   └── VPN wait durations:
|       ├── Initial duration: 6s
|       └── Additional duration: 5s
├── Shadowsocks server settings:
|   └── Enabled: no
├── HTTP proxy settings:
|   └── Enabled: no
├── Control server settings:
|   ├── Listening address: :8000
|   └── Logging: yes
├── OS Alpine settings:
|   ├── Process UID: 0
|   ├── Process GID: 0
|   └── Timezone: europe/london
├── Public IP settings:
|   ├── Fetching: every 12h0m0s
|   └── IP file path: /tmp/gluetun/ip
└── Version settings:
    └── Enabled: yes
2023-06-14T09:55:44+01:00 INFO using existing username root corresponding to user id 0
2023-06-14T09:55:44+01:00 INFO [routing] default route found: interface eth0, gateway 172.18.0.1, assigned IP 172.18.0.3 and family v4
2023-06-14T09:55:44+01:00 DEBUG [routing] ip rule add from 172.18.0.3/32 lookup 200 pref 100
2023-06-14T09:55:44+01:00 INFO [routing] adding route for 0.0.0.0/0
2023-06-14T09:55:44+01:00 DEBUG [routing] ip route replace 0.0.0.0/0 via 172.18.0.1 dev eth0 table 200
2023-06-14T09:55:44+01:00 INFO [firewall] setting allowed subnets...
2023-06-14T09:55:44+01:00 INFO [routing] default route found: interface eth0, gateway 172.18.0.1, assigned IP 172.18.0.3 and family v4
2023-06-14T09:55:44+01:00 DEBUG [routing] ip rule add to 172.18.0.0/16 lookup 254 pref 98
2023-06-14T09:55:44+01:00 INFO [dns over tls] using plaintext DNS at address 1.1.1.1
2023-06-14T09:55:44+01:00 INFO [http server] http server listening on [::]:8000
2023-06-14T09:55:44+01:00 INFO [healthcheck] listening on 127.0.0.1:9999
2023-06-14T09:55:44+01:00 INFO [firewall] allowing VPN connection...
2023-06-14T09:55:44+01:00 DEBUG [firewall] iptables --append OUTPUT -d 212.102.35.199 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT
2023-06-14T09:55:44+01:00 DEBUG [firewall] iptables --append OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:44+01:00 DEBUG [firewall] ip6tables-nft --append OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:44+01:00 INFO [openvpn] OpenVPN 2.5.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov  2 2022
2023-06-14T09:55:44+01:00 INFO [openvpn] library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
2023-06-14T09:55:44+01:00 INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]212.102.35.199:1194
2023-06-14T09:55:44+01:00 INFO [openvpn] UDP link local: (not bound)
2023-06-14T09:55:44+01:00 INFO [openvpn] UDP link remote: [AF_INET]212.102.35.199:1194
2023-06-14T09:55:50+01:00 INFO [healthcheck] program has been unhealthy for 6s: restarting VPN (see https://github.com/qdm12/gluetun/wiki/Healthcheck)
2023-06-14T09:55:50+01:00 INFO [vpn] stopping
2023-06-14T09:55:50+01:00 INFO [vpn] starting
2023-06-14T09:55:50+01:00 INFO [firewall] allowing VPN connection...
2023-06-14T09:55:50+01:00 DEBUG [firewall] iptables --delete OUTPUT -d 212.102.35.199 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT
2023-06-14T09:55:50+01:00 DEBUG [firewall] iptables --delete OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:50+01:00 DEBUG [firewall] ip6tables-nft --delete OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:50+01:00 DEBUG [firewall] iptables --append OUTPUT -d 143.244.42.69 -o eth0 -p udp -m udp --dport 1194 -j ACCEPT
2023-06-14T09:55:50+01:00 DEBUG [firewall] iptables --append OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:50+01:00 DEBUG [firewall] ip6tables-nft --append OUTPUT -o tun0 -j ACCEPT
2023-06-14T09:55:50+01:00 INFO [openvpn] OpenVPN 2.5.8 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov  2 2022
2023-06-14T09:55:50+01:00 INFO [openvpn] library versions: OpenSSL 3.1.1 30 May 2023, LZO 2.10
2023-06-14T09:55:50+01:00 INFO [openvpn] TCP/UDP: Preserving recently used remote address: [AF_INET]143.244.42.69:1194
2023-06-14T09:55:50+01:00 INFO [openvpn] UDP link local: (not bound)
2023-06-14T09:55:50+01:00 INFO [openvpn] UDP link remote: [AF_INET]143.244.42.69:1194
2023-06-14T09:55:50+01:00 INFO [openvpn] [nl-ams-v095.prod.surfshark.com] Peer Connection Initiated with [AF_INET]143.244.42.69:1194
2023-06-14T09:55:51+01:00 ERROR [openvpn] Unrecognized option or missing or extra parameter(s) in [PUSH-OPTIONS]:7: block-outside-dns (2.5.8)
2023-06-14T09:55:51+01:00 INFO [openvpn] TUN/TAP device tun0 opened
2023-06-14T09:55:51+01:00 INFO [openvpn] /sbin/ip link set dev tun0 up mtu 1500
2023-06-14T09:55:51+01:00 INFO [openvpn] /sbin/ip link set dev tun0 up
2023-06-14T09:55:51+01:00 INFO [openvpn] /sbin/ip addr add dev tun0 10.8.8.27/24
2023-06-14T09:55:51+01:00 INFO [openvpn] Initialization Sequence Completed
2023-06-14T09:55:51+01:00 INFO [dns over tls] downloading DNS over TLS cryptographic files
2023-06-14T09:55:51+01:00 INFO [healthcheck] healthy!
2023-06-14T09:55:52+01:00 INFO [dns over tls] downloading hostnames and IP block lists
2023-06-14T09:55:56+01:00 INFO [dns over tls] init module 0: validator
2023-06-14T09:55:56+01:00 INFO [dns over tls] init module 1: iterator
2023-06-14T09:55:56+01:00 INFO [dns over tls] start of service (unbound 1.17.1).
2023-06-14T09:55:56+01:00 INFO [dns over tls] generate keytag query _ta-4a5c-4f66. NULL IN
2023-06-14T09:55:56+01:00 INFO [dns over tls] generate keytag query _ta-4a5c-4f66. NULL IN
2023-06-14T09:55:57+01:00 INFO [dns over tls] ready
2023-06-14T09:55:57+01:00 INFO [vpn] You are running on the bleeding edge of latest!
2023-06-14T09:55:57+01:00 INFO [ip getter] Public IP address is 143.244.42.70 (Netherlands, North Holland, Amsterdam)
alpine:~#

Share your configuration

version: "3"

services:
  gluetun:
    image: qmcgaw/gluetun:latest
    container_name: gluetun
    cap_add:
      - NET_ADMIN
    devices:
      - /dev/net/tun:/dev/net/tun
    ports:
      - 8888:8888/tcp # HTTP proxy
      - 8388:8388/tcp # Shadowsocks
      - 8388:8388/udp # Shadowsocks
      - 8080:8080
      - 6881:6881
      - 6881:6881/udp
    volumes:
      - './gluetun:/gluetun'
    environment:
      - VPN_SERVICE_PROVIDER=surfshark
      - VPN_TYPE=openvpn 
      - OPENVPN_USER=x
      - OPENVPN_PASSWORD=x
      - SERVER_COUNTRIES=Netherlands
      - LOG_LEVEL=debug
      - PUID=0
      - PGID=0
      - TZ=Europe/London

  qbittorrent:
    image: lscr.io/linuxserver/qbittorrent:latest
    container_name: qbittorrent
    environment:
      - PUID=0
      - PGID=0
      - TZ=UTC
      - WEBUI_PORT=8080
      - ANONYMOUS_MODE=true
      - LOG_LEVEL=debug
    network_mode: "service:gluetun"
    volumes:
      - "./qbittorrent/config:/config"
      - "./media/downloads:/downloads"
    restart: unless-stopped
bnhf commented 1 year ago

@Lumino2

I highly doubt this is a gluetun problem. I run gluetun plus 12 other Docker containers in a Proxmox CT w/4GB of RAM. Resource usage is very low, and performance is excellent with Speedtest.net results through the VPN typically in the 500-600Mbps both up and down. I'm running Debian 11.6 as the Proxmox CT OS.

screencapture-pve-8006-2023-06-14-09_11_38

Lumino2 commented 1 year ago

Im also running debian 11 on a fresh proxmox installation with 1 VM. Speed is not an issue. I've ran other docker containers and memory was stable. As soon as I start the gluetun container and run some traffic through it, the memory skyrockets. Capture2

bnhf commented 1 year ago

@Lumino2

There's really no reason to run gluetun in a Proxmox VM -- from a resource usage standpoint it's much better to run it in a Proxmox CT. Also, Alpine is great for building small footprint Docker containers, but I think Debian is a better choice for a Proxmox container.

So, my suggestion is rather than trying to figure out where the issue lies with your inefficient current setup -- why not spin-up a Proxmox CT with Docker? 2 CPUs and 4GB of RAM should be sufficient. There's one small change you need to make to the Proxmox .conf file for any OpenVPN or Wireguard use in a Proxmox CT, but I detailed that here: https://github.com/qdm12/gluetun/discussions/1482

bnhf commented 1 year ago

@Lumino2

Since it's been a few months since I tried gluetun in a Proxmox VM rather than a CT, I spun-up a VM running Debian 11.7 with Docker, gluetun and the linuxserver.io Firefox container. I ran a few Speedtests, and other traffic through the VPN with no indication of a memory leak or overflow (2 CPUs and 8GB RAM). The Firefox container requires more resources than most, but ran fine. As mentioned in my previous comment, this is not a resource efficient setup, but other than that I saw no issues.

If there's some reason you really want to use a VM rather than a CT, I'd look to your use of Alpine as the OS, or qBittorrent as likely culprits.

Lumino2 commented 1 year ago

Thanks for the effort, I tried firefox and memory usage was fine. I conclude that it's a qbittorrent issue. Will try the more efficient setup you recommended and thus this can be closed. Thanks!

qdm12 commented 1 year ago

I conclude that it's a qbittorrent issue.

Are you running both in the same container? You can always run a shell or exec in a container and call top to see which process holds memory. On alpine you could use htop with apk add htop; htop

agneevX commented 1 year ago

I've had luck disabling unbound and Docker health check.

l1meju1ce commented 1 year ago

How did you disable the healthcheck? I thought it couldn't be disabled so I just pointed mine to localhost with "HEALTH_TARGET_ADDRESS=127.0.0.1:9999"

agneevX commented 1 year ago

In docker compose:

services:
  gluetun:
    container_name: gluetun2
    image: qmcgaw/gluetun
    restart: unless-stopped
    healthcheck:
      disable: true
l1meju1ce commented 1 year ago

In docker compose:

services:
  gluetun:
    container_name: gluetun2
    image: qmcgaw/gluetun
    restart: unless-stopped
    healthcheck:
      disable: true

Thank you, I thought it was an environment variable to make gluetun itself stop doing a healthcheck, I didn't know that docker could stop it.

Lumino2 commented 1 year ago

For everyone reading this thread, i invested more time in this and resolved the issue. I set up a dedicated Docker container for qbittorrent & gluetun, allocating 4GB of RAM with ballooning enabled. While Proxmox's interface indicates full RAM usage, checking via Proxmox's terminal and HTOP shows only 4GB of RAM is actually in use.