search

hashicorp / nomad-driver-lxc

HashiCorp Nomad LXC driver plugin
Mozilla Public License 2.0
31 stars 19 forks source link

Driver does not release memory after the container is destroyed #20

Closed aroundthfur closed 2 years ago

aroundthfur commented 4 years ago

I have noticed that the driver never releases the memory (RAM) taken for each allocation, after the allocation (e.g. LXC Container) is destroyed and GC-ed. This leads to a very high memory consumption after a while, since the driver just accumulates the memory it "needs".

I suspect this is not a 'normal' behaviour, but I have not yet found the place where this should be fixed/implemented in the code. Comparing the Docker driver with LXC, it seems to me that the same actions are taken in the DestroyTask(taskID string, force bool) error (1) callback.

Did anyone else experience this issue so far?

My environment:

$ apt-cache policy lxc
lxc:
  Installed: 1:3.1.0+really3.0.3-8
  Candidate: 1:3.1.0+really3.0.3-8
  Version table:
 *** 1:3.1.0+really3.0.3-8 500
        500 http://deb.debian.org/debian buster/main amd64 Packages
        100 /var/lib/dpkg/status
$ cat /etc/debian_version 
10.2
$ uname -a
Linux data01 4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u2 (2019-11-11) x86_64 GNU/Linux

Any advice would be appreciated. Also if you need more information from my side, let me know!

(1): https://nomadproject.io/docs/internals/plugins/task-drivers/#destroytasktaskid-string-force-bool-error

towe75 commented 4 years ago

Possible this would fix your problem: https://github.com/pascomnet/nomad-driver-lxc/commit/d13ba27c3447e64a888fbef4ecc2600e86169cd9

You can try to compile and test this forked version. Be aware that it contains some other changes as well. I would, of course, contribute/port some of the changes if it helps you.

aroundthfur commented 4 years ago

Thanks! I tried this but after some further testing i realized that the issue is not just releasing memory after the task is destroy/stopped, but rather that during a lifetime of the container, the driver keeps increasing memory usage.

After a while this ends up being a lot and i am not sure what is the driver actually using the memory for.

Did you ever encounter such behaviour?

towe75 commented 4 years ago

Sorry, no. We're not using it in production. I would assume that it maybe leaks some bytes in the periodical stats collector.

aroundthfur commented 4 years ago

Ahh yes, thank you for the hint! I will investigate this and report back if I manage to find something.