Open VoidJuiceConcentrate opened 1 month ago
@qdm12 is more or less the only maintainer of this project and works on it in his free time. Please:
Commenting to add: It's this error that seemingly causes the vpn to restart,
2024-10-01T18:42:37Z ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 9.9.9.9:53: read udp 10.2.0.2:46851->9.9.9.9:53: i/o timeout
This error is actively being thrown when other containers are able to upload/download from VPN. This causes gluetun to restart.
Hello there,
First of all, there are a few i/o dns erros with the server 12.159.2.159
, try removing the quadrant provider from your list of upstream resolvers? It seems to be an issue here.
Commenting to add: It's this error that seemingly causes the vpn to restart,
Ok so that's a bit strange, the getting public IP address information
should only be run once when the VPN connection got setup, so I doubt other containers would be able to download/upload at the same time. It also does not affect the VPN auto healing feature, you can set LOG_LEVEL=debug
to see more healthcheck information in the logs regarding auto healing. All in all, it might a problem with DNS: perhaps try just setting the resolvers to Google?? Perhaps both quadrant and quad9 are not working well (9.9.9.9:53) 🤷
@qdm12 yeah i'll change back to cloudflare, but I was getting the same errors on cloudflare. Quad9 appeares to just error out less than cloudflare.
Also, thanks for the tip! I'll add that in and see what it shows.
2024-10-01T19:40:27Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1, assigned IP 172.17.0.5 and family v4
2024-10-01T19:40:27Z DEBUG [routing] ip rule add from 172.17.0.5/32 lookup 200 pref 100
2024-10-01T19:40:27Z INFO [routing] adding route for 0.0.0.0/0
2024-10-01T19:40:27Z DEBUG [routing] ip route replace 0.0.0.0/0 via 172.17.0.1 dev eth0 table 200
2024-10-01T19:40:27Z INFO [firewall] setting allowed subnets...
2024-10-01T19:40:27Z INFO [routing] default route found: interface eth0, gateway 172.17.0.1, assigned IP 172.17.0.5 and family v4
2024-10-01T19:40:27Z DEBUG [routing] ip rule add to 172.17.0.0/16 lookup 254 pref 98
2024-10-01T19:40:27Z INFO TUN device is not available: open /dev/net/tun: no such file or directory; creating it...
2024-10-01T19:40:27Z INFO [dns] using plaintext DNS at address 1.1.1.1
2024-10-01T19:40:27Z INFO [http server] http server listening on [::]:8000
2024-10-01T19:40:27Z INFO [healthcheck] listening on 127.0.0.1:9999
2024-10-01T19:40:27Z DEBUG [wireguard] Wireguard server public key: 6viKMPw7x82HUJJbPTA08M3oY9U0SRDxpktohKENJTk=
2024-10-01T19:40:27Z DEBUG [wireguard] Wireguard client private key:
2024-10-01T19:40:27Z DEBUG [wireguard] Wireguard pre-shared key: [not set]
2024-10-01T19:40:27Z INFO [firewall] allowing VPN connection...
2024-10-01T19:40:27Z DEBUG [firewall] /sbin/iptables --append OUTPUT -d 37.19.200.22 -o eth0 -p udp -m udp --dport 51820 -j ACCEPT
2024-10-01T19:40:27Z DEBUG [firewall] /sbin/iptables --append OUTPUT -o tun0 -j ACCEPT
2024-10-01T19:40:27Z DEBUG [firewall] /sbin/ip6tables --append OUTPUT -o tun0 -j ACCEPT
2024-10-01T19:40:27Z INFO [wireguard] Using available kernelspace implementation
2024-10-01T19:40:27Z INFO [wireguard] Connecting to x.x.x.x:51820
2024-10-01T19:40:27Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-10-01T19:40:27Z INFO [dns] downloading hostnames and IP block lists
2024-10-01T19:40:27Z INFO [healthcheck] healthy!
2024-10-01T19:40:34Z DEBUG [healthcheck] unhealthy: dialing: dial tcp4: lookup cloudflare.com: i/o timeout
2024-10-01T19:40:40Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2024-10-01T19:40:40Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-10-01T19:40:40Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-10-01T19:40:40Z INFO [vpn] stopping
2024-10-01T19:40:40Z ERROR [vpn] getting public IP address information: context canceled
2024-10-01T19:40:40Z ERROR [vpn] cannot get version information: Get "https://api.github.com/repos/qdm12/gluetun/commits": context canceled
2024-10-01T19:40:40Z DEBUG [wireguard] closing controller client...
2024-10-01T19:40:40Z DEBUG [wireguard] removing IPv4 rule...
2024-10-01T19:40:40Z DEBUG [wireguard] shutting down link...
2024-10-01T19:40:41Z DEBUG [wireguard] deleting link...
2024-10-01T19:40:41Z INFO [vpn] starting
2024-10-01T19:40:41Z DEBUG [wireguard] Wireguard server public key: 6viKMPw7x82HUJJbPTA08M3oY9U0SRDxpktohKENJTk=
2024-10-01T19:40:41Z DEBUG [wireguard] Wireguard client private key: MEs...Vo=
2024-10-01T19:40:41Z DEBUG [wireguard] Wireguard pre-shared key: [not set]
2024-10-01T19:40:41Z INFO [firewall] allowing VPN connection...
2024-10-01T19:40:41Z INFO [wireguard] Using available kernelspace implementation
2024-10-01T19:40:41Z INFO [wireguard] Connecting to x.x.x.x:51820
2024-10-01T19:40:41Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-10-01T19:40:42Z WARN [dns] cannot update filter block lists: context deadline exceeded (Client.Timeout or context cancellation while reading body)
2024-10-01T19:40:42Z INFO [dns] attempting restart in 10s
2024-10-01T19:40:51Z ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 1.1.1.1:53: read udp 10.2.0.2:56096->1.1.1.1:53: i/o timeout
2024-10-01T19:40:52Z INFO [dns] downloading hostnames and IP block lists
2024-10-01T19:40:53Z INFO [healthcheck] program has been unhealthy for 11s: restarting VPN
2024-10-01T19:40:53Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-10-01T19:40:53Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-10-01T19:40:53Z INFO [vpn] stopping
2024-10-01T19:40:53Z DEBUG [wireguard] closing controller client...
2024-10-01T19:40:53Z DEBUG [wireguard] removing IPv4 rule...
2024-10-01T19:40:53Z DEBUG [wireguard] shutting down link...
2024-10-01T19:40:53Z DEBUG [wireguard] deleting link...
2024-10-01T19:40:53Z INFO [vpn] starting
2024-10-01T19:40:53Z DEBUG [wireguard] Wireguard server public key: 6viKMPw7x82HUJJbPTA08M3oY9U0SRDxpktohKENJTk=
2024-10-01T19:40:53Z DEBUG [wireguard] Wireguard client private key: MEs...Vo=
2024-10-01T19:40:53Z DEBUG [wireguard] Wireguard pre-shared key: [not set]
2024-10-01T19:40:53Z INFO [firewall] allowing VPN connection...
2024-10-01T19:40:53Z INFO [wireguard] Using available kernelspace implementation
2024-10-01T19:40:53Z INFO [wireguard] Connecting to 37.19.200.22:51820
2024-10-01T19:40:53Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
2024-10-01T19:41:03Z ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 1.1.1.1:53: read udp 10.2.0.2:59629->1.1.1.1:53: i/o timeout
2024-10-01T19:41:07Z WARN [dns] cannot update filter block lists: Get "https://raw.githubusercontent.com/qdm12/files/master/malicious-hostnames.updated": context deadline exceeded (Client.Timeout exceeded while awaiting headers), Get "https://raw.githubusercontent.com/qdm12/files/master/malicious-ips.updated": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2024-10-01T19:41:07Z INFO [dns] attempting restart in 20s
Also, it's doing this every so often:
2024-10-01T19:46:04Z ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 127.0.0.1:53: server misbehaving
Mostly it's a bunch of this
2024-10-01T20:06:56Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:34168->1.0.0.1:853: i/o timeout
2024-10-01T20:06:56Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:51764->149.112.112.112:853: i/o timeout
2024-10-01T20:06:56Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:56590->12.159.2.159:853: i/o timeout
2024-10-01T20:06:56Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:51772->149.112.112.112:853: i/o timeout
2024-10-01T20:06:57Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:40504->12.159.2.159:853: i/o timeout
2024-10-01T20:06:57Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:51886->8.8.4.4:853: i/o timeout
2024-10-01T20:06:58Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:50786->8.8.8.8:853: i/o timeout
2024-10-01T20:06:58Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:50796->8.8.8.8:853: i/o timeout
2024-10-01T20:06:58Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:50802->8.8.8.8:853: i/o timeout
2024-10-01T20:07:00Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:57386->9.9.9.9:853: i/o timeout
2024-10-01T20:07:02Z WARN [dns] exchanging over DoT connection: read tcp 10.2.0.2:51890->8.8.4.4:853: i/o timeout
At this moment, the other containers are using the VPN fine until gluetun decides to reboot itself again.
Turn off DoT, and it will work. At least, that fixed the problem for me.
Thanks for the input! Unfortunately I would rather have my dns lookups happen over TLS as it's an extra layer of security. So, ideally I would like DOT to stay on and work with my connection.
Update: It's happening through OpenVPN as well, and the server is restarting because of it there also.
2024-10-04T23:05:36Z DEBUG [healthcheck] unhealthy: dialing: dial tcp4: lookup cloudflare.com: i/o timeout
2024-10-04T23:05:42Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
2024-10-04T23:05:42Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
2024-10-04T23:05:42Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
2024-10-04T23:05:42Z INFO [vpn] stopping
2024-10-04T23:05:42Z ERROR [vpn] getting public IP address information: context canceled
2024-10-04T23:05:42Z ERROR [vpn] cannot get version information: Get "https://api.github.com/repos/qdm12/gluetun/commits": context canceled
I'm trying with a lower MTU but if that doesn't work I'm gonna bust out Wireshark because it's starting to sound like a network issue.
Hi,
Same issue with ProtonVPN...
Trying many things during all the day and finally found the solution : https://github.com/qdm12/gluetun/issues/2442#issuecomment-2418687189
Adding - HEALTH_TARGET_ADDRESS=127.0.0.1:9999
solving the problem for me.
Edit : Don't work finally : ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 127.0.0.1:53: server misbehaving
Same issue here:
gluetun | 2024-10-31T18:21:52Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:39302->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:21:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:51294->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:21:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:39314->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:21:56Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:51304->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:21:56Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:39330->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:22:32Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:38746->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:22:34Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:37450->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:22:34Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:37464->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:22:36Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:38762->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:22:36Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:37474->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:39Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:57100->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:39Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:57104->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:41Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:50342->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:41Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:50352->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:43Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:57118->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:43Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:50364->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:48Z WARN [dns] dial tcp 1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:48Z WARN [dns] dialing dns over tls server: dial tcp 1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:48Z WARN [dns] dial tcp 1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:48Z WARN [dns] dialing dns over tls server: dial tcp 1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:52Z INFO [healthcheck] program has been unhealthy for 6s: restarting VPN
gluetun | 2024-10-31T18:23:52Z INFO [healthcheck] 👉 See https://github.com/qdm12/gluetun-wiki/blob/main/faq/healthcheck.md
gluetun | 2024-10-31T18:23:52Z INFO [healthcheck] DO NOT OPEN AN ISSUE UNLESS YOU READ AND TRIED EACH POSSIBLE SOLUTION
gluetun | 2024-10-31T18:23:52Z INFO [vpn] stopping
gluetun | 2024-10-31T18:23:52Z INFO [port forwarding] stopping
gluetun | 2024-10-31T18:23:52Z INFO [firewall] removing allowed port 33622...
gluetun | 2024-10-31T18:23:52Z INFO [port forwarding] removing port file /tmp/gluetun/forwarded_port
gluetun | 2024-10-31T18:23:52Z INFO [vpn] starting
gluetun | 2024-10-31T18:23:52Z INFO [firewall] allowing VPN connection...
gluetun | 2024-10-31T18:23:52Z INFO [wireguard] Using available kernelspace implementation
gluetun | 2024-10-31T18:23:52Z INFO [wireguard] Connecting to 190.2.131.156:51820
gluetun | 2024-10-31T18:23:52Z INFO [wireguard] Wireguard setup is complete. Note Wireguard is a silent protocol and it may or may not work, without giving any error message. Typically i/o timeout errors indicate the Wireguard connection is not working.
gluetun | 2024-10-31T18:23:52Z INFO [healthcheck] healthy!
gluetun | 2024-10-31T18:23:52Z WARN [dns] dial tcp 1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:52Z WARN [dns] dialing dns over tls server: dial tcp 1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:52Z WARN [dns] dial tcp 1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:52Z WARN [dns] dialing dns over tls server: dial tcp 1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:55750->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:36770->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:55762->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:54Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:55778->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:56Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:55786->1.1.1.1:853: i/o timeout
gluetun | 2024-10-31T18:23:56Z WARN [dns] exchanging over dns over tls connection: read tcp 10.2.0.2:36778->1.0.0.1:853: i/o timeout
gluetun | 2024-10-31T18:23:56Z ERROR [vpn] getting public IP address information: fetching information: Get "https://ipinfo.io/": dial tcp: lookup ipinfo.io on 127.0.0.1:53: server misbehaving
gluetun | 2024-10-31T18:23:56Z INFO [port forwarding] starting
gluetun | 2024-10-31T18:23:56Z INFO [port forwarding] gateway external IPv4 address is 190.2.131.159
gluetun | 2024-10-31T18:23:56Z INFO [port forwarding] port forwarded is 51688
gluetun | 2024-10-31T18:23:56Z INFO [firewall] setting allowed input port 51688 through interface tun0...
gluetun | 2024-10-31T18:23:56Z INFO [port forwarding] writing port file /tmp/gluetun/forwarded_port
@VoidJuiceConcentrate Try with DOT=off? I doubt it would resolve it really 🤔 At least just to narrow down the issue. The fact it says healthy!
and then fails is rather strange... I would also be curious if anyone finds a solution (for this particular problem, not for any unhealthy error!)
One more thing, when you say:
The VPN will work from the perspective of the containers running through it: i can use curl to grab the public IP from each container and it'll resolve DNS, talk with the server, and return the VPN's public IP. However, gluetun will throw I/O timeouts and TLS errors, especially around connecting with DOT or github for blocklist updates.
Are you doing any bandwidth intensive data transfers as soon as Gluetun establishes the VPN connection? That might explain the timeouts 🤔
Is this urgent?
No
Host OS
No response
CPU arch
aarch64
VPN service provider
ProtonVPN
What are you using to run the container
docker-compose
What is the version of Gluetun
Running version latest built on 2024-09-29T18:12:41.313Z (commit 67ae5f5)
What's the problem 🤔
The connection is spotty, unsure how else to describe. So, the connection will appear to be fine, gluetun will connect and show public IP, and forward the welcome message from the VPN itself. The VPN will work from the perspective of the containers running through it: i can use curl to grab the public IP from each container and it'll resolve DNS, talk with the server, and return the VPN's public IP. However, gluetun will throw I/O timeouts and TLS errors, especially around connecting with DOT or github for blocklist updates.
It stabilizes to just io timeout warnings (instead of errors) after erroring out and auto-restarting 3 or 4 times. Seemingly, the third or fourth restart of services is able to make a DOT connection, connect to github over TLS, and pull down blocklists fine.
I wonder if there's some sort of time to wait between the Wireguard service connecting, and the actual VPN being ready for data flow?
Either way, below is my configuration. Specifying TZ, a reduced MTU, and quad9 as the first DOT provider has helped a bit, it's brought incessant restarting down to a handful of restarts before it connects, then a restart every 30 minutes when DOT TLS errors start piling up.
While gluetun is going through all this, the rest of my containers going through gluetun is able to use the VPN absolutely fine (when it's not in the middle of an auto-heal restart).
Ideally, I would like to boot gluetun in an extra-verbose mode so I can see more about what's going on. Anyway, I hope I provided enough information for diagnostics!
Share your logs (at least 10 lines)
Share your configuration