Mongod Memory usage controls

[hooper303]

I have a small home network and running 5.6.29

For whatever reason, mongod just grows and grows in memory usage. It is usually above 1.2 GB after a day or so. Is anyone else seeing this and have thoughts on how to fix. My temporary workaround is to limit the memory usage of the container but as you can imagine it results in the process being killed and re-spawned every day, I haven't had this issue until the recent 5.6 builds.

I was thinking maybe an environment variable to pass along to the mongod init script?

[sphen13]

ive got the same question. just testing this out in docker and yeah it seems to grow out of control. i was running this outside of docker on an equally sized vm with no issue.

[sundowndk]

Same here, every 2-3 days I need to restart the container, since MongoDB just eats up all memory and then I think docker kills the process and it never starts backup. And as sphen13 mentioned Unifi ran perfectly outside of docker on a VM before trying out docker.

I also tried 'unifi.xms' and 'unifi.xmx' settings that are mentioned on the Ubnt forums, but it had no effect.

[sphen13]

too early to tell - but i have just limited the amount of ram the container uses - so far so good. i am sure it depends on your environment and how many devices are in your controller - but what i added to docker run:

--memory="500m" --memory-swap="2g"

[sundowndk]

I tried the memory limits as well (easy to set when using portainer), but it does not help.

I've got 4 unifi servers running on the same VM, with 20GB of mem, within 5-6 days the two largest servers (100+ clients each) will have maxed out the memory and are half way though swap. This eventually result in everything grinding to a halt.

If I add the memory constrains on the container, problems start happening when the container hits 80-90% memory usage. The logs says that MongoDB has crashed because it could not allocate memory and its never restarted, leaving the controller useless.

I am thinking of reverting back to using a single VM for each controller without docker, but updating and backup of data is much easier with docker, so hopefully somebody can figure this out.

I might try an external MongoDB server before going backup to single VMs.

[jacobalberty]

I can't seem to reproduce this on any of my systems running either 5.7.x or 5.6.x. I just updated the beta tag on the hub from mongodb 3.2 to 3.4 though, try running it use the jacobalberty/unifi:beta image, you'll also need to specify a pkgurl environment variable use https://dl.ubnt.com/unifi/5.6.30/unifi_sysvinit_all.deb for 5.6.30, you can add -e PKGURL=https://dl.ubnt.com/unifi/5.6.30/unifi_sysvinit_all.deb to your docker command line if you're running it on the command line

The next stable candidates are still slated to be on mongodb 3.2 but if this is a fix I can go ahead and update those before 5.6.35 and 5.7.19 go stable.

[sundowndk]

I will give it a spin tonight and report back. Thanks!

[sundowndk]

I will give it a spin tonight and report back. Thanks!

[sundowndk]

Tried the beta version and I get this error over and over in the mongodb.log upon starting the container.

`2018-02-20T21:06:38.792+0000 I - [initandlisten] Fatal assertion 28579 UnsupportedFormat: Unable to find metadata for table:index-0-8850994146936012738 Index: {name: id, ns: ace.account} - version too new for this mongod. See http://dochub.mongodb.org/core/3.4-index-downgrade for detailed instructions on how to handle this error. at src/mongo/db/storage/wiredtiger/wiredtiger_index.cpp 241 2018-02-20T21:06:38.793+0000 I - [initandlisten]

***aborting after fassert() failure`

Dont know why its talking about downgrading when version is clearly 3.4.13

2018-02-20T21:06:42.920+0000 I CONTROL [initandlisten] MongoDB starting : pid=5346 port=27117 dbpath=/usr/lib/unifi/data/db 64-bit host=dc94a5230ce1 2018-02-20T21:06:42.920+0000 I CONTROL [initandlisten] db version v3.4.13

[sundowndk]

Ok, problem was again memory. Still had the restrictions on the container and upgrade to 3.4 db took a massive 8.5GB og memory to complete.

Server is running now, will report back on memory consumption tomorrow.

[sundowndk]

Server has now been running for almost 2 days, memory consumptions is at 4GB total, which is 30% of available memory for the container.

I have restartet the server this morning with a lower memory restriction (4GB total), will report back tonight with usage.

[sundowndk]

So the server has now been running with a memory restriction of 4GB for little over 24 hours and memory usage is at 99.3%. Server still responds and I have not seen any "out of memory" messages in the logs.

I don't see any improvement in memory usage when using mongodb 3.4, but it does seam to run better when it reaches container limits.

[sphen13]

hey @sundowndk - not to compare projects here - but since i was running into the same issue i switched to https://hub.docker.com/r/linuxserver/unifi/ as a test. my memory and overall load is way lower using this. worth a try?

picture - can you tell when i switched containers? :)

[sundowndk]

Thanks @sphen13, I will give that image a spin and see if I get the same results. The difference is pretty clear in the picture. Wonder why, difference in JavaVM version?

[jacobalberty]

Thats weird that the linuxserver version would be any different, its the exact same version of mongodb (same source and everything) and openjdk version as the beta images

[sundowndk]

@jacobalberty Is the beta openjdk version different from the stable version?

At the moment I have 4 Unifi servers running on a single VM, one of them I have running on the Beta with mongodb 3.4, and it does seam to run better even tho its at 99% memory (4GB) usage. The other 3 run the stable version.

I just had to restart 2 out of the 3, because mongodb had failed because of memory shortage, but as of now the beta version has been running for 2 days and it still responds. The ones that was just restarted had been up for a little over 4 days, and was using 99% as well, along with a boat load of swap.

[jacobalberty]

The stable version has mongodb 3.2 iirc while beta is mongodb 3.4, both are on openjdk 8 for the major version but there may be some minor version differences as stable is on debian beta is on ubuntu.

[sphen13]

Interesting. Yeah not sure what the linuxserver container exactly does since I couldn’t see the Dockerfile too early to tell if it will stay stable

[jacobalberty]

I do see one change between linuxserver and mine, linuxserver calls java directly instead of using jsvc, just now rebuilt beta on the hub to use java directly.

[sundowndk]

I will give it a try and let you know if I see any changes.

[sphen13]

fyi - tried the beta container:

/usr/local/bin/docker-entrypoint.sh: line 135: java: command not found

[jacobalberty]

@sphen13 did you add -e PKGURL=https://dl.ubnt.com/unifi/5.6.30/unifi_sysvinit_all.deb to your launch command line?

[sundowndk]

A little update:

I have tried the linuxserver container and it ends up doing exactly the same thing, no change at all.

So I started digging around for memory issues with MongoDB and I found this config setting:

https://docs.mongodb.com/manual/reference/configuration-options/#storage.wiredTiger.engineConfig.cacheSizeGB

It specifically mentions running MongoDB inside Docker. From what I can gather is that Mongo allocates 50% of available memory - 1GB for cache. The problem is that its using the full amount of memory of the underlying VM, in my case 15GB, so that would be something like 6.5GB, way more than the 3.5 I allowed the container to use. Eventually it starts using swap and with 4 containers doing this, swap will end up at 100% as well and then issues start happening.

I added the --wiredTigerCacheSizeGB option via unifi.db.extraargs in the system.properties of the controller like this:

unifi.db.extraargs=--wiredTigerCacheSizeGB\=1

The containers have been running for a couple of hours now and they seam to stay at around 1.7GB of memory, where before they would be at 2.5 - 3GB within 30 minutes.

Don't know if this fixes the issue, but its worth a try..I will report back later.

[sphen13]

great feedback @sundowndk

[sundowndk]

@sphen13 Thx.

The containers have been running fine for little over a day, the two large controllers (100+ units) have settled around 2.5gb - 2.7gb of used memory. The smaller controllers have been stable at 1.4GB and 2GB since yesterday. No swap is in use.

Gonna try and lower the memory available for the containers to see how they perform with less than 3GB of memory. Ubnt says that the controller should run fine with just 2GB, but I have noticed back when I ran the controller directly on the VM that there where issues with create backups with large retention. What I found there was that the database dump was kept in /run before being compressed and placed in the DL folder. And since the /run folder is a tmpfs drive there where not enough space to complete the task.

The MongoDB docs on the cache option said that it could be configured thru a cgroup as well, maybe its something @jacobalberty should look into and make an ENV that could be set. I presume it could be done via the /etc/mongodb.conf as well. Its defiantly something that needs to be set as default, especially if the container has a memory constraint or there are multiple controllers running on the same machine.

[jacobalberty]

it might be possible to install cgroup-bin in the container then use cgget -n --values-only --variable memory.limit_in_bytes / to get the memory limit and have the startup script modify the value to some appropriate fraction of the total limit if you set a memory limit. Mongodb i thought is already supposed to be checking that value so it shouldnt be running into that issue but maybe Im misunderstanding the documentation for mongodb, or maybe its not intelligent enough about it to leave enough ram for unifi to run as well.

[FuzzyMistborn]

Any update on this? My very small controller (15-20 devices) is eating about 5gb of memory. Seems out of whack.

[edif30]

Yes I would like to know if there has been any other experiences or ideas? I just restarted my container and viewing with docker stats and top you can watch mongod just grow over time. Yesterday my container was eating up 4G. After a restart it begins around 550MB and just grows to about 2.5G in under 24 hours. By the time it settles... mongod is eating up a majority of the memory.

[sundowndk]

I have had a long standing issue with one of my controllers, it runs for around 2 days and then I get this in the logs:

Exception in thread "inform-110" Exception in thread "http-nio-8880-ClientPoller-1" java.lang.OutOfMemoryError: Java heap space

And the server becomes unresponsive. VM has plenty of memory and only this one server has the issue. I have resorted to restarting the container every two days, but a permanent fix besides moving it to its own VM would be greate.

Funny thing is that its not even the controller with the most devices connected (around 30-40). The largest of the 6 controllers has more than 200 devices, and never gets a reboot unless a new version comes along. The other 4 are similar in size to the one have problems and they run without issues.

[abel408]

What workaround would be the cleanest solution for this issue? Just found mongod with over 20GB on my server...

[ecdlguy]

Is there a workaround yet? I have the controller running on my Syno with 10GB of RAM and in a few weeks I need to find another way to run the controller because I'm out of RAM (Only 4 Unifi Devices!) I did "compact database" but this led to an even higher RAM usage... cheers, Thorsten

[ziptbm]

I set a memory limit in the Docker settings for this container on Synology at 1GB and I haven't run into any issues since then.

[ecdlguy]

@ziptbm ok I didn't try that because @sundowndk wrote:

I tried the memory limits as well (easy to set when using portainer), but it does not help.

But since it's working for you I also turned on a limit at 1GB. At least for the first 30 minutes it seems to work. cheers, Thorsten

[ecdlguy]

Just wanted to confirm that enabling a memory limit for the container solves the problem for me. I did not notice any negative side effects. cheers, thorsten

[Looooopy]

Have you guys checked the drive swap so it does not intensively access the disk instead?

[ecdlguy]

I don't see swap getting larger on the host after enabling the memory limit.

[Elfe]

I added "command: --wiredTigerCacheSizeGB 0.25" into my docker-compose.yml for the mongo service. This limits mongo to 256MB of storage cache.

[maisun]

Just wanted to confirm that enabling a memory limit for the container solves the problem for me. I did not notice any negative side effects. cheers, thorsten

How do you limit memory? I tried with
--memory="2g" --memory-swap="2g" but still run out of memory everyday

[PrplHaz4]

I seem to have gotten this under control (with the jacobalberty image). Not sure what finally did it, but it's a bit strange that the actual memory usage seems to be around the two limits combined. I'm pretty confident the leak has something to do with a certain combo of settings, but haven't seen much work done to figure out what exactly that might be.

• Unifi Controller v5.14.23
• Running on USG3P @ 4.4.51.5287926
• "New" UI
• DPI On
• L2TP VPN w/Radius Auth
• Minimal Port Forwarding
• 2 SSIDs (incl one Guest)

Relevant Settings:

# docker-compose.yml
image: jacobalberty/unifi:5.14.23
environment:
      - JVM_MAX_HEAP_SIZE=2048M

# unifi/data/system.properties
unifi.db.extraargs=--wiredTigerCacheSizeGB\=2

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[Elfe]

I think there might be multiple issues.

This issue is about MongoDB which is written in C++. So no JVM settings apply.

What would be the best default value of wiredTigerCacheSizeGB for most users? (It uses the greater value of "50% RAM - 1GB" or "0.25GB" as default) Any host running more than 2.5GB of memory will put more resources into the MongoDB instance :(

diff --git a/docker-compose.yml b/docker-compose.yml
index 3589dfb..10bc0f8 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -2,6 +2,7 @@ version: '2.3'
 services:
   mongo:
     image: mongo:3.6
+    command: --wiredTigerCacheSizeGB 0.25
     container_name: ${COMPOSE_PROJECT_NAME}_mongo
     networks:
       - unifi

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[Elfe]

This is not stale.

[lockhartcn]

is there any solutions? my unifi/data/db increse to 17G now ....-,-!!

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.