Open engageub opened 5 months ago
github-actions[bot]
commented
5 months ago @qdm12 is more or less the only maintainer of this project and works on it in his free time. Please:
elmagow
commented
5 months ago Same for me on latest Fedora. Not as high usage, but still on top of my cpu usage with my server doing nothing
qdm12
commented
5 months ago OpenVPN version: 2.6 is the difference. Try using OPENVPN_VERSION=2.5?
engageub
commented
5 months ago Hi,
Thank you for the response. Yes, OPENVPN_VERSION=2.5 reduces the CPU similar to v3.37.0. However, there is a problem with consistency in the latest version.
When I run the command sudo docker stats <gluetun_container_name> for about a minute to get the stats, the CPU utilization suddenly spikes to 100% and comes back to normal, whereas with v3.37.0 this is not the case.
Thank you
qdm12
commented
5 months ago reduces the CPU similar to v3.37.0.
Do you also have the problem with v3.38.0?
Anyway if Openvpn 2.6 is at fault, there isn't much I can do as far as I know, nothing changed except the openvpn version. Still a strange issue... It might be worth reporting it to the OpenVPN dev team? 🤔
the CPU utilization suddenly spikes to 100% and comes back to normal, whereas with v3.37.0 this is not the case.
I'm not sure I understand this fully, the CPU spikes to 100% for Gluetun only, or for the entire machine, and for how long? Does it happen only when querying docker stats?
engageub
commented
5 months ago Do you also have the problem with v3.38.0?
v3.38.0 is slightly better than v3.37.0 when compared to memory usage. v3.38.0 was consuming about 56 MB where as v3.37.0 was consuming about 64 MB. CPU is normal in this version.
Anyway if Openvpn 2.6 is at fault, there isn't much I can do as far as I know, nothing changed except the openvpn version. Still a strange issue... It might be worth reporting it to the OpenVPN dev team? 🤔
If OpenVPN version is the only problem, then it is supposed to be informed to them to resolve the issue.
I'm not sure I understand this fully, the CPU spikes to 100% for Gluetun only, or for the entire machine, and for how long? Does it happen only when querying
docker stats?
I started the container with --cpus=1 options in 4 core machine. The CPU is 100% only for gluetun container displayed by docker stats command.
This can also be tested on Play with Docker website directly without using --cpus option.
engageub
commented
4 months ago In v3.37.0, I also observed that the default health checks were consuming more CPU. A 5-second interval ping is too aggressive for health checks. By disabling health checks and removing the exposed ports, CPU utilization has drastically reduced. I was only able to run 100 containers earlier, which reached about 90% CPU utilization. After removing health checks and deleting the ports, 100 containers are now using only about 10% CPU in total. This shows a significant variation in CPU usage with health checks enabled. Memory utilization has also dropped below 20 MB for each container after removing health checks and exposed ports.
Thank you
qdm12
commented
3 months ago By removing the exposed ports
Do you mean by removing the EXPOSE instruction in the Dockerfile?
qdm12
commented
3 months ago So after some investigation on my side: the Docker healthcheck runs Gluetun in "health client mode", and this spikes the CPU usage because it loads the program (not even because of what the program actually does). For me it goes from 0% to 2% usage, so not a big deal. Possibly your CPU is low power hence this "100% usage" spike. If you want you can disable the docker healthcheck as you did. I tried running the healthcheck without any code (just return nil) and it would still spike to 1.9%, so that confirms it. Disabling it actually does not disable the internal healthcheck, that still works fine without using resources. This healthcheck has been like this for a very long time now, so this is not a regression; a good observation to note in the wiki though, if someone wants to run many Gluetun containers (I'm adding a performance document in the faq directory of the wiki).
Now, additional questions:
By removing the exposed ports: Do you mean by removing the EXPOSE instruction in the Dockerfile? Exposed ports are for documentation only. You must be mistaken as to its performance impact.A 5-second interval ping is too aggressive for health checks. I disagree, having it every 10 minutes would make 100 Gluetun instances eat all the CPU power every 10 minutes, which is also annoying. The better choice is to leave it to 5 seconds (fast feedback) for normal usage (i.e. 2 Gluetun) and to fully disable it if you need to run many Gluetuns (see above)engageub
commented
3 months ago Thank you for your response.
I have downloaded the Dockerfile, removed the health checks and exposed ports, and subsequently rebuilt the Docker image. From my observations, the primary contributor to CPU usage in version 3.37.0 is the health checks, as evidenced by the process statistics obtained from the top command. I have also eliminated the exposed ports since they are not necessary for my use case.
The CPU that we can use in common to get the similar result is play with docker. It is available for free and I ran the commands on that host.
It is possible that users who previously employed gluetun did not use it with a large number of containers, and therefore might not have noticed the increased CPU utilization.
Below are the responses to your queries:
I removed the EXPOSE instruction and health checks from the Dockerfile as they are not required for my application. However, I did not observe a significant impact on performance as a result of these changes.
The health check process is consuming more CPU than any other process within the container, including the VPN itself. It is unclear whether the high CPU usage is due to how data is fetched, the way the program loads, or any logging mechanisms used. This issue may be exacerbated by slow internet speeds, as the CPU usage remains consistent with a 5-second interval, given that both timeout and interval are set to 5 seconds. With around 200 containers running, even a slight increase in CPU usage per container can lead to substantial overall consumption. Using the --no-healthcheck option in the Docker run command disables health checks.
The following table provides CPU consumption details, as observed using the docker stats command on Play with Docker with no CPU limits applied. Health checks were disabled using the --no-healthcheck option in the Docker run command:
| Gluetun Version | CPU with Health check | CPU without Health check | OpenVPN Version |
|---|---|---|---|
| v3.37.0 | Reaches 8% | ~0.02 - 0.5% | 2.5 |
| v3.38.0 | Reaches 8% | ~0.02 - 0.5% | 2.5 |
| v3.38.0 | Reaches 10% | ~0.18 - 2% | 2.6 |
You may need to contact OpenVPN support to investigate why there is a fourfold increase in CPU usage compared to the previous version.
Thank you
qdm12
commented
3 months ago It is possible that users who previously employed gluetun did not use it with a large number of containers, and therefore might not have noticed the increased CPU utilization.
Yes totally (my case too). I'm writing up a performance document focused on running many instances and how to reduce CPU and memory usage.
The health check process is consuming more CPU than any other process within the container, including the VPN itself. It is unclear whether the high CPU usage is due to how data is fetched, the way the program loads, or any logging mechanisms used. This issue may be exacerbated by slow internet speeds, as the CPU usage remains consistent with a 5-second interval, given that both timeout and interval are set to 5 seconds
It's not the actual internal Gluetun health check and autohealing doing the TCP dialing every 5 seconds. That part uses pretty much zero CPU; it's running the Gluetun binary that uses CPU, I believe because it's a "fat" 23MB binary and that takes some processing to load. I even tried just doing "nothing" when running gluetun in "health client mode", and it would still use CPU every 5 seconds, showing it's not the actual code responsible for it, it's more the system loading the fat binary program to execute it. I might actually replace it with a shell script since it's specific to Docker, and that would remove the CPU usage spikes.
You may need to contact OpenVPN support to investigate why there is a fourfold increase in CPU usage compared to the previous version.
That's odd indeed. Can you try running v3.39.0 with openvpn 2.6 to see if it changes anything? v3.38.0 was running with 2.6.8, whereas v3.39.0 uses openvpn 2.6.11
engageub
commented
3 months ago Thank you for the response. I used version v3.39.0 without health checks and it consumes about 160% CPU. This is more than 80 times the previous versions.
Thank you
qdm12
commented
2 months ago Oh my god, openvpn what the hell. This is under bandwidth stress right? And if you revert to using openvpn 2.5, it goes back to "normal" right?
For the healthcheck, it's pretty much doomed: the CPU usage spikes to at least 0.9% when running any command, even /bin/true which does nothing. It's just the start process overhead. Launching an empty Go program uses 1%, and launching Gluetun uses 2% (because of all the code imports here and there). I could setup a separate program with less imports to reduce that 2% to 1%, but really I think it doesn't help much, because if you want to run many Gluetuns, you're better off turning off the healthcheck within Docker. Thoughts on that?
engageub
commented
2 months ago Hi, The CPU utilization figures provided for version 3.39.0 are reported without bandwidth stress. I have not used openVPN 2.5 on 3.39.0. At the moment, play with docker has diskspace issues so can't run commands there. For comparison, I have used OpenVPN 2.5 with versions 3.37.0 and 3.38.0, and observed that CPU consumption was minimal. Detailed CPU utilization data for these versions is presented in the tabular format in the previous comments.
Please note there are existing issues concerning increased CPU consumption by health checks. For more information, please refer to the links below. One proposed solution is to disable health checks and implement retry mechanisms directly within the service.
Thank you
engageub
commented
2 months ago I would like to provide an update regarding my previous comment. I have successfully downgraded the OpenVPN version from 2.6.11 to 2.5 on version 3.39.0, and observed that the CPU usage has returned to normal levels, consistent with those of the previous versions listed in the table.
It appears that version 3.39.0 may pose challenges for users, as the default OpenVPN version 2.6.11 is exhibiting unusually high CPU consumption (160%) even under normal conditions without significant bandwidth stress.
Please find the updated table below with the latest version.
| Gluetun Version | CPU with Health check | CPU without Health check | OpenVPN Version |
|---|---|---|---|
| v3.37.0 | Reaches ~8% | ~0.02 - 0.5% | 2.5 |
| v3.38.0 | Reaches ~8% | ~0.02 - 0.5% | 2.5 |
| v3.39.0 | Reaches ~8% | ~0.02 - 0.5% | 2.5 |
| v3.38.0 | Reaches ~10% | ~0.18 - 2% | 2.6.8 |
| v3.39.0 | Reaches ~170% | ~160% | 2.6.11 |
In Internet Income script, I am utilizing version v3.37.0 and would prefer not to upgrade unless there are clear benefits compared to previous versions.
I have provided all the necessary information. Please proceed with the next steps as required.
Thank you
qdm12
commented
2 months ago One proposed solution is to disable health checks and implement retry mechanisms directly within the service.
Indeed, that's already done sort of 😉 The vpn gets auto healed within the container, and the internal healthcheck runs every period no matter what is defined in the Docker image.
I have successfully downgraded the OpenVPN version from 2.6.11 to 2.5 on version 3.39.0, and observed that the CPU usage has returned to normal levels, consistent with those of the previous versions listed in the table. I am utilizing version v3.37.0 and would prefer not to upgrade unless there are clear benefits compared to previous versions.
Why not use v3.39 with OPENVPN_VERSION=2.5 though? There are quite a few fixes and features, added see the release notes.
It appears that version 3.39.0 may pose challenges for users, as the default OpenVPN version 2.6.11 is exhibiting unusually high CPU consumption (160%) even under normal conditions without significant bandwidth stress.
The odd thing is, on my machines (WSL host and arch linux host), running Gluetun v3.39 + Openvpn 2.6 (default) uses between 0% and 0.1% of CPU when idling. I'm starting to think this is related to your OS/kernel 🤔
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
8e0d270a72ea gluetun 0.00% 481.7MiB / 9.711GiB 4.84% 5.97MB / 374kB 0B / 0B 37Have you tried for example https://github.com/kylemanna/docker-openvpn to see if it makes a difference? Since it's using alpine:latest, I would guess it's using openvpn 2.6 as well.
I'm trying to narrow down which part if at fault:
I'm inclined to think it's 1. right now 🤔
engageub
commented
2 months ago Thank you for the response.
Why not use v3.39 with
OPENVPN_VERSION=2.5though? There are quite a few fixes and features, added see the release notes.
Yes, openvpn 2.5 with v3.39.0 can be added as the CPU Utilization is normal with this combination.
Have you tried for example https://github.com/kylemanna/docker-openvpn to see if it makes a difference? Since it's using
alpine:latest, I would guess it's using openvpn 2.6 as well.
This is OpenVPN server and not client. I don't see options to use this docker image as openVPN client.
I'm trying to narrow down which part if at fault:
- your OS/kernel <-> openvpn 2.6
- openvpn 2.6: so far, unlikely since it's fine on my machines)
- Gluetun setting an openvpn option that makes openvpn 2.6 misbehave, but is fine with openvpn 2.5
As previously noted, I am using play with docker, which is a free version anyone can use by one click sign-in.
The operating system is utilizing the kernel version 4.4.0-210-generic, which is relatively outdated. This could potentially contribute to the issue. However, the specific differences between versions 2.5 and 2.6 that might be influencing this situation remain unclear.
telnetdoogie
commented
2 days ago Although I've seen high CPU on occasion from gluetun, the past week or so it's really hitting hard right now... I tried disabling the docker healthcheck but that doesn't seem to make much difference.
I have switched to using OpenVPN 2.5 as suggested above, but that doesn't seem to make any difference for me. Been using gluetun for over a year with the same stack but for some reason just lately it seems to be overrunning things...
Not sure how to troubleshoot in a meaningul / helpful way so I'm all ears if anyone has any ideas about how to identify where the bottleneck is in a scientific way.
Running on a Synology NAS with docker 27.3.1 Connected to VyprVPN via OpenVPN (how I wish they'd get wireguard configs going for linux) Two containers behind - transmission, and prowlarr... The above screenshots are with no traffic, no active downloads.
Nothing but goodness in the gluetun logs:
Is this urgent?
None
Host OS
Ubuntu 22
CPU arch
x86_64
VPN service provider
OPENVPN
What are you using to run the container
docker run
What is the version of Gluetun
Running version latest built on 2024-05-18T18:08:57.405Z (commit 4218dba)
What's the problem 🤔
When the docker image qmcgaw/gluetun is used. The CPU utilization of the container goes to about 100% of 1 CORE where as the following image qmcgaw/gluetun:v3.37.0 uses less than 1% of 1 CORE. Could you please look into the latest version and compare it with v3.37.0.
Share your logs (at least 10 lines)
Share your configuration