Feature request: Restarting both openvpn and dns if it stays unhealthy for N seconds

redtripleAAA commented 3 years ago

What's the feature? 🧐 Restarting both openvpn and dns if it stays unhealthy for N seconds?

Support this new feature because that and that It will help many users with different VPN providers, especially in this thread, where it seems many people they have to restart manually as autoheal container is not doing what is supposed to do.

Optional extra information 🚀 Example: https://github.com/MarkusMcNugen/docker-qBittorrentvpn/issues/60

Where in that thread, talking about qbittorrentVPN + AutoHeal container

so it would be nicer if gluetun, if would have a built-in auto-heal feature within the container to restart itself after # of seconds, and we can configure that with env maybe or if it detects any errors per the logs (so we don't have to worry about configuring any env)

Slack chat thread: https://qdm12.slack.com/archives/CQQ5BQU8N/p1613421956000600

Thanks

qdm12 commented 3 years ago

There is a test image qmcgaw/gluetun:restart-unhealthy where openvpn gets restarted when the container is unhealthy, feel free to test it out 😉

redtripleAAA commented 3 years ago

Awesome! Deploying it now on a secondary stack and testing with netdata container.

redtripleAAA commented 3 years ago

I got some random restarts which is good, but not sure why.

Here is the latest logs

2021/04/10 16:45:52 INFO storage: merging by most recent 7350 hardcoded servers and 7350 servers read from /gluetun/servers.json
2021/04/10 16:45:52 INFO routing: default route found: interface eth0, gateway 192.168.0.1
2021/04/10 16:45:52 INFO routing: local ethernet link found: eth0
2021/04/10 16:45:52 INFO routing: local subnet found: 192.168.0.0/20
2021/04/10 16:45:52 INFO routing: default route found: interface eth0, gateway 192.168.0.1
2021/04/10 16:45:52 INFO routing: adding route for 0.0.0.0/0
2021/04/10 16:45:52 INFO firewall: firewall disabled, only updating allowed subnets internal list
2021/04/10 16:45:52 INFO routing: default route found: interface eth0, gateway 192.168.0.1
2021/04/10 16:45:52 INFO openvpn configurator: checking for device /dev/net/tun
2021/04/10 16:45:52 WARN TUN device is not available: open /dev/net/tun: no such file or directory
2021/04/10 16:45:52 INFO openvpn configurator: creating /dev/net/tun
2021/04/10 16:45:52 INFO firewall: enabling...
2021/04/10 16:45:52 INFO firewall: enabled successfully
2021/04/10 16:45:52 INFO healthcheck: listening on 127.0.0.1:9999
2021/04/10 16:45:52 INFO dns over tls: using plaintext DNS at address 1.1.1.1
2021/04/10 16:45:52 INFO http server: listening on 0.0.0.0:8000
2021/04/10 16:45:52 INFO firewall: setting VPN connection through firewall...
2021/04/10 16:45:52 INFO openvpn configurator: starting openvpn
2021/04/10 16:45:52 INFO openvpn: OpenVPN 2.4.10 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Jan  4 2021
2021/04/10 16:45:52 INFO openvpn: library versions: OpenSSL 1.1.1k  25 Mar 2021, LZO 2.10
2021/04/10 16:45:52 INFO openvpn: CRL: loaded 1 CRLs from file [[INLINE]]
2021/04/10 16:45:52 INFO openvpn: TCP/UDP: Preserving recently used remote address: [AF_INET]154.13.1.50:1197
2021/04/10 16:45:52 INFO openvpn: UDP link local: (not bound)
2021/04/10 16:45:52 INFO openvpn: UDP link remote: [AF_INET]154.13.1.50:1197
2021/04/10 16:45:53 INFO openvpn: [berlin419] Peer Connection Initiated with [AF_INET]154.13.1.50:1197
2021/04/10 16:45:54 INFO openvpn: OpenVPN ROUTE6: OpenVPN needs a gateway parameter for a --route-ipv6 option and no default was specified by either --route-ipv6-gateway or --ifconfig-ipv6 options
2021/04/10 16:45:54 INFO openvpn: OpenVPN ROUTE: failed to parse/resolve route for host/network: 2000::/3
2021/04/10 16:45:54 INFO openvpn: TUN/TAP device tun0 opened
2021/04/10 16:45:54 INFO openvpn: /sbin/ip link set dev tun0 up mtu 1500
2021/04/10 16:45:54 INFO openvpn: /sbin/ip addr add dev tun0 10.2.110.7/24 broadcast 10.2.110.255
2021/04/10 16:45:54 WARN openvpn: OpenVPN was configured to add an IPv6 route over tun0. However, no IPv6 has been configured for this interface, therefore the route installation may fail or may not work as expected.
2021/04/10 16:45:54 INFO openvpn: Initialization Sequence Completed
2021/04/10 16:45:54 INFO VPN routing IP address: 154.13.1.50
2021/04/10 16:45:54 INFO dns over tls: downloading DNS over TLS cryptographic files
2021/04/10 16:45:55 INFO healthcheck: healthy!
2021/04/10 16:46:07 INFO dns over tls: downloading hostnames and IP block lists
2021/04/10 16:46:09 INFO dns over tls: init module 0: validator
2021/04/10 16:46:09 INFO dns over tls: init module 1: iterator
2021/04/10 16:46:09 INFO dns over tls: start of service (unbound 1.10.1).
2021/04/10 16:46:09 INFO dns over tls: generate keytag query _ta-4a5c-4f66. NULL IN
2021/04/10 16:46:12 INFO dns over tls: ready
2021/04/10 16:46:13 INFO There is a new release v3.15.0 (v3.15.0) created 43 days ago
2021/04/10 16:46:13 INFO VPN gateway IP address: 10.2.110.1
2021/04/10 16:46:13 INFO port forwarding: Found persistent forwarded port data for port 35256
2021/04/10 16:46:13 INFO port forwarding: Forwarded port data expires in 61 days
2021/04/10 16:46:13 INFO port forwarding: Port forwarded is 35256 expiring in 61 days
2021/04/10 16:46:13 INFO port forwarding: Writing port to / /volume1/docker/gluetun/gluetun-restart/config/port-forwarding/port.conf
2021/04/10 16:46:13 ERROR port forwarding: open / /volume1/docker/gluetun/gluetun-restart/config/port-forwarding/port.conf: no such file or directory
2021/04/10 16:46:13 INFO firewall: setting allowed input port 35256 through interface tun0...
2021/04/10 16:46:18 INFO ip getter: Public IP address is 154.13.1.50 (Germany, Berlin, Berlin)
2021/04/10 18:45:56 INFO dns over tls: generate keytag query _ta-4a5c-4f66. NULL IN
2021/04/10 20:35:41 WARN Caught OS signal terminated, shutting down
2021/04/10 20:35:41 WARN ip getter: context canceled: exiting loop
2021/04/10 20:35:41 INFO ip getter: Removing ip file /tmp/gluetun/ip
2021/04/10 20:35:41 WARN dns over tls: context canceled: exiting loop
2021/04/10 20:35:41 WARN openvpn: context canceled: exiting loop
2021/04/10 20:35:41 INFO Clearing forwarded port status file / /volume1/docker/gluetun/gluetun-restart/config/port-forwarding/port.conf
2021/04/10 20:35:41 WARN healthcheck: context canceled: shutting down server
2021/04/10 20:35:41 WARN http server: context canceled: shutting down
2021/04/10 20:35:41 WARN http server: shut down
2021/04/10 20:35:41 WARN healthcheck: server shut down
2021/04/10 20:35:41 WARN dns over tls: loop exited
2021/04/10 20:35:41 WARN openvpn: loop exited
2021/04/10 20:35:41 INFO firewall: removing allowed port 35256 through firewall...
2021/04/10 20:35:42 ERROR remove / /volume1/docker/gluetun/gluetun-restart/config/port-forwarding/port.conf: no such file or directory
2021/04/10 20:35:42 WARN ip getter: loop exited
2021/04/10 20:35:43 ERROR port forwarding: cannot remove allowed port 35256 through interface tun0: failed executing "iptables --delete INPUT -i tun0 -p tcp --dport 35256 -j ACCEPT": : context deadline exceeded
2021/04/10 20:35:43 WARN port forwarding: loop exited
2021/04/10 20:35:43 INFO Shutdown successful

qdm12 commented 3 years ago

The container should not restart, only Openvpn from within should be restarted without the program exiting at all, so there is something wrong going on.

Maybe keep the logs of previous containers? (you can configure that in docker-compose.yml)

qdm12 commented 3 years ago

Cool! For the logs you can actually simply find the container ID (using the Synology docker UI or docker ps) and the log file of the container and its previous instances will all be together in /var/lib/docker/containers/<container-id>/<container-id>-json.log

redtripleAAA commented 3 years ago

Nice. What do you recommend to change/add in the docker compose in gluetun restart in order to map the logs in a folder outside the container.

---
version: '2.4'
services:
  gluetun:
    image: qmcgaw/gluetun:restart-unhealthy
    container_name: gluetun-restart
    environment:
      - PUID=0
      - PGID=0
      - TZ=America/Toronto
      - VPNSP=private internet access
      - REGION=DE Berlin #Config files from PIA from here https://www.privateinternetaccess.com/helpdesk/kb/articles/where-can-i-find-your-ovpn-files-2
      - PORT_FORWARDING=on #Complete https://github.com/qdm12/gluetun/wiki/Environment-variables
      - PORT_FORWARDING_STATUS_FILE=/gluetun/port-forwarding/port.conf
      - OPENVPN_USER=p####### #Change to YOUR Username
      - OPENVPN_PASSWORD=############ #Change to YOUR Password
    volumes:
      - /volume1/docker/gluetun/gluetun-restart/config:/gluetun
    ports:
      - 8001:8000 #HTTP Server https://github.com/qdm12/gluetun/wiki/HTTP-Control-server#OpenVPN
      - 19999:19999 #Netdata
    cap_add:
      - NET_ADMIN
    restart: unless-stopped

I use https://dozzle.dev/ but it's only good for realtime debugging

qdm12 commented 3 years ago

I don't think you actually can after having a look at their docs, sorry for misleading that. So you cam either:

Just check the file in the /var/lib/docker directory for that container id
Setup Loki and Promtail. That's a fancy addition to have your logs collected and stored, accessible through a nice UI. That's on my TODO list so I can't help that much yet 😄

redtripleAAA commented 3 years ago

For gluetun restart container

I see the following

root@Synology:~# docker exec -it 89ab026e8ef6 /bin/sh
/ # cd /var/lib
/var/lib # ls -1
apk
arpd
ip6tables
iptables
misc
udhcpd

For netdata connected to the VPN container

I see

root@Synology:~# docker exec -it c4d16fa99986 /bin/sh
/ #
/ #
/ # cd /var/lib
/var/lib # ls -1
apk
arpd
ip6tables
iptables
libvirt
misc
netdata
nut
udhcpd

I don't see "docker" in "lib" as you mentioned. Not sure if I am missing something

qdm12 commented 3 years ago

Oh you need to find it on your host 😉 I think you can use the Synology terminal App in their UI as I recall; otherwise you would have to SSH in there.

For example this command should show it in the console: cat /var/lib/docker/containers/<container-id>/<container-id>-json.log or you could copy it somewhere else with cp /var/lib/docker/containers/<container-id>/<container-id>-json.log /volume1/somepath/logs.log and access it with the file explorer.

redtripleAAA commented 3 years ago

I think I found the logs location, it's a bit differnet from other docker hosts platforms

root@Synology:/volume1/@docker/containers#
root@Synology:/volume1/@docker/containers/89ab026e8ef6aa1964bc402724d5e5ef46a48cec6efa53d1465944f1f963342d# ls -1
checkpoints
config.v2.json
hostconfig.json
hostname
hosts
log.db
mounts
resolv.conf
resolv.conf.hash

And I noticed it's the same logs from the GUUI of the DSM Synology

Next Steps: I will keep monitoring and share with you the logs on the next crash

qdm12 commented 3 years ago

Any restart so far 🧐?

qdm12 commented 3 years ago

Please re-pull that image :restart-unhealthy as the newer one contains a bunch of bug fixes and improvements. It should now detect when it's unhealthy within 1 to 5 seconds and should restart openvpn. If it's still unhealthy on the second try, it doubles the wait time from 6 to 12s and so on. I think that version should be good to go but I'll let you run it for a few days 😉

redtripleAAA commented 3 years ago

Believe or not, since I deployed the testing container, both were working fine. :s

Pulling both of them now and updating the containers with the new updated images.

Thank you for the headsup!

redtripleAAA commented 3 years ago

It's been up since I deployed the original and the test image. I think your trick helped a lot here and we should maybe close?

qdm12 commented 3 years ago

Glad to hear! Yeah let's close it. Although note that this only restarts openvpn, not the DNS server... although the DNS server is unlikely to fail I guess.

qdm12 / gluetun

Feature request: Restarting both openvpn and dns if it stays unhealthy for N seconds #386