search

pducharme / UniFi-Video-Controller

Docker for Unifi-Video Controller (Ubiquiti Networks)
199 stars 105 forks source link

3.10.1 extreme memory and CPU usage #128

Closed squirtbrnr closed 5 years ago

squirtbrnr commented 5 years ago

I just updated to :latest which is 3.10.1 and the memory usage is easily 5-10x what it used to be. As soon as the container starts up the memory usage quickly spikes to about 2.1Gb usage and CPU usage jumps to 50-70%. Only after a few minutes does the memory usage drop down to about 1.98Gb and then 1.8Gb usage but it stays there and CPU usage never goes below 30%. I downgraded and used the :3.9.11 tag (since there is no :3.9.12 tag) and the memory usage is "normal" at about 200Mb minimum and no more than 500Mb maximum and CPU usage at about 6%. Even starting with a new container using :latest the memory and CPU usage is extremely high. If I go into the system configuration of the UniFi Video controller via the web page, it tells me there is medium memory usage (yellow bar) and 4.6 of 8.2Gb used. When running 3.9.x, the system configuration tells me it's low memory usage (green bar) and 2.9Gb of 8.2Gb used.

Lastly, if I let the controller container run for more than an hour, the container will crash. The only message I can see in the logs around the same time refer to MongoDB and either multiple errors trying to connect or database failure. This also occurs on a new container and new configuration (new linked volumes).

squirtbrnr commented 5 years ago

Additionally, I get a few Java runtime errors in the docker container logs.

kevsterd commented 5 years ago

On mine, I get a constant "waiting for mongo to start" forever. Ill try a few things but suspect its foobar...

Ill try rolling back to 3.9.x

kevsterd commented 5 years ago

Rolling back seems to work. I suspect its an iffy release. Anyone else tried it ?

squirtbrnr commented 5 years ago

I'm thinking something major was changed in the video controller from UniFi. I have a separate installation that uses the Debian packages to run as a service and it too had a huge spike in memory and CPU usage. When I went to the webpage it was stuck on "Upgrading, almost finished". I let it run for nearly 24 hours and it never finished and the machine was virtually locked up.

kevsterd commented 5 years ago

I recon so also. This distribution certainly dosent get the comments like the old ctindel docker image does it. I'm supprised its not been a bigger issue.

kevsterd commented 5 years ago

Looking at the 3.10.x release notes, it appears 3.10.x has a pre-req of 3.9.12 before upgrade to 3.10.x to upgrade some components relating to OpenJDK.....

Hmmm. Anyone else had any luck ?

squirtbrnr commented 5 years ago

I was on 3.9.12 before attempting to upgrade. The :latest tag was at 3.9.12 before it was updated to 3.10.1. No specific :3.9.12 tag was ever created for this image.

AddoSolutions commented 5 years ago

I cannot even get it to come up using :latest with a completely clean install. Not sure what the deal is with it.

squirtbrnr commented 5 years ago

Neither can I. New container, new volume links, and it gets stuck loading the database.

stefangordon commented 5 years ago

In my case it spawned so many java instances (~32000) it maxed out the system threads and crippled the machine - was an ordeal just to recover the host.

Switching back to the 3.9.12 tag got things up and running again. Docs say you have to do the upgrade via the web UI once first to go to 3.10.... will investigate.

fryfrog commented 5 years ago

You can use -e JVM_MX=2048M to get the memory usage under control. The forum post points out that even Ubiquity set that to something like 1500M in their own device.

squirtbrnr commented 5 years ago

I could never get my container to start with that option enabled. I don’t remember the error it gave but it would just restart. Haven’t tried it again in a while.

EDIT: I'm confusing my personal docker install with my other debian install as a service at work. The JVM option didn't work for my debian install and just caused the service to restart. Haven't tried it on my personal container though.

fryfrog commented 5 years ago

I think the thread says it defaults to some percentage of your memory, maybe start big and slowly decrease until it starts up properly?

fryfrog commented 5 years ago

https://forums.unraid.net/topic/35393-support-pducharmes-dockers-support-thread/?page=21&tab=comments#comment-725138

squirtbrnr commented 5 years ago

I just created a new container with new volume links and it started with no problems. I restored my backup from my 3.9.12 container (I believe this is the recommended upgrade path anyway) and it restored no problem. I still have some extreme memory and CPU usage issues, but the JVM_MAX variable does work. Unfortunately the UniFi video software (specifically Java) is determined to use every last bit of memory you allocate for the container. Not a docker container issue anymore as I have a working 3.10.1 container, this is now a UniFi issue.