qdm12 / gluetun

VPN client in a thin Docker container for multiple VPN providers, written in Go, and using OpenVPN or Wireguard, DNS over TLS, with a few proxy servers built-in.
https://hub.docker.com/r/qmcgaw/gluetun
MIT License
7.8k stars 364 forks source link

Feature request: traffic based healthcheck #1856

Open dfadev opened 1 year ago

dfadev commented 1 year ago

What's the feature 🧐

Make the healthcheck optionally monitor packet counts to determine if a connection is still healthy.

Extra information and references

On a saturated connection, the healthcheck can fail but the interface is still passing packets.

gluetun8  | 2023-09-10T06:46:16-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T07:22:16-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T08:47:20-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:14:43-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:42:06-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T10:48:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:01:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:08:39-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:16:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:30:09-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:39:52-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T11:49:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:00:05-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:05:05-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:12:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:18:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:25:06-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:30:48-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:42:29-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T12:43:34-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:03:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:09:09-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T13:50:17-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:02:53-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:12:44-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:17:49-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:22:30-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:35:59-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:47:51-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T14:57:49-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:09:35-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:17:11-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:21:27-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:32:24-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:37:58-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:47:26-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:53:46-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T15:59:42-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:08:55-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:13:47-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:21:47-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:22:52-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:38:54-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:47:37-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T16:48:42-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:02:17-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:07:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:12:52-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:14:41-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:18:59-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:20:05-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:21:25-04:00 INFO [healthcheck] program has been unhealthy for 1m10s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:26:14-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:32:32-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:39:38-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:47:43-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T17:55:35-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:01:37-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:07:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:08:08-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:14:04-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:21:03-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:26:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:27:18-04:00 INFO [healthcheck] program has been unhealthy for 1m5s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:39:10-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:48:48-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T18:59:13-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
gluetun8  | 2023-09-10T19:05:01-04:00 INFO [healthcheck] program has been unhealthy for 1m0s: restarting VPN (see https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md)
qdm12 commented 1 year ago

It's not because packets get sent through the interface that the connection is working right? Or are you talking about receiving packets? What's your use case to monitor packets versus the current tcp dialing to 1.1.1.1:443?

dfadev commented 1 year ago

It's for torrenting over links with high latency and lots of bandwidth management, like cellular. When there are a high number of peers the healthcheck begins to fail even though traffic is still being exchanged.

For a quick POC, I added this patch:

diff --git a/internal/healthcheck/health.go b/internal/healthcheck/health.go
index 2cd61426..8aa1b3ac 100644
--- a/internal/healthcheck/health.go
+++ b/internal/healthcheck/health.go
@@ -5,6 +5,7 @@ import (
        "errors"
        "fmt"
        "net"
+       "os"
        "time"
 )

@@ -63,6 +64,10 @@ func (s *Server) runHealthcheckLoop(ctx context.Context, done chan<- struct{}) {
 }

 func (s *Server) healthCheck(ctx context.Context) (err error) {
+       if os.Getenv("DISABLE_HEALTH_TCP") == "true" {
+               return nil
+       }
+
        // TODO use mullvad API if current provider is Mullvad

        address, err := makeAddressToDial(s.config.TargetAddress)

And use this for the healthcheck in docker-compose:

  test: ["CMD-SHELL", "packet_count=$$(cat /sys/class/net/tun0/statistics/rx_packets); sleep 2; new_packet_count=$$(cat /sys/class/net/tun0/statistics/rx_packets); if [ ! -z $$new_packet_count ] && [ $$new_packet_count -le $$packet_count ]; then exit 1; fi"]

With DISABLE_HEALTH_TCP set to true, and the custom healthcheck command set, I can push the connection to about 7500 peers without the VPN restarting at all. Without, the TCP check starts to recycle the VPN when the peer count reaches about 2500.

Restarting the VPN is not ideal because it takes time to handshake that many connections and unnecessary because some peer connections are still able to exchange traffic.

qdm12 commented 2 months ago

This will eventually be added thanks to #2411 which parses iptables results, and gets packets/bytes sent for each chain rule. We can use that to monitor traffic.