Running out of Memory

[jamesmr89]

I have to restart the container about once a day, with this in the logs:

Exception in thread "inform-88" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform-340" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-42" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform-13" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-100" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform-253" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-32" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-55" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform-210" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-56" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-58" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-51" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-53" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform-258" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portals-191" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-109" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-198" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-54" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-bio-8843-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portals-13" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-181" Exception in thread "portals-190" java.lang.OutOfMemoryError: GC overhead limit exceeded

java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-81" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portals-16" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-129" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-134" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-142" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-62" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-46" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-57" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-68" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-49" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-77" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-108" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-82" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-61" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-13" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-78" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-15" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-112" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-21" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-84" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-120" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-202" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-66" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-64" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "webapi-205" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-208" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "http-bio-8443-Acceptor-0" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-65" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-59" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-63" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "inform_stat-69" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-139" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-123" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-148" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-120" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-137" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-121" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-135" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "WsHeartbeat" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-240" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "WebsocketWriteThread" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "cluster-3-127.0.0.1:27117" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-132" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "Ws" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-206" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception in thread "portal-156"

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "portal-156"

What do you expect to happen?

I'm wondering if there are any way we can toggle some of the memory settings via environmental vars for xms and xmx in the startup script?

I'm running on a machine with 16gigs of memory, but I'm not sure what toggles I should be playing with here to troubleshoot, obviously this is more of an application issue than container, but want to see if we can expose more memory related settings?

[jacobalberty]

There is an environment variable to control the -Xms environment variable, however it is currently being cleared by the entrypoint script so you won't be able to pass it right now. I'll make a few changes to expose those and put it in the beta tag. I'll post back when that's done

[jacobalberty]

Ok, if you use jacobalberty/unifi:beta (you'll need to pass the right PKGURL for the version you use) you can now use the environment variables JVM_MAX_HEAP_SIZE (Default is 1024M) to set -Xmx and JVM_INIT_HEAP_SIZE (no default) to set -Xms .

If you haven't used the beta tag before then just a note it installs the package at PKGURL at run time instead of at build time so it's a little slower on the first docker up, latest stable is 5.7.23 so to run that docker up -e PKGURL=https://dl.ubnt.com/unifi/5.7.23/unifi_sysvinit_all.deb jacobalberty/unifi:beta . Add more -e options for JVM_MAX_HEAP_SIZE and JVM_INIT_HEAP_SIZE to suit your needs, if it works then let me know and I'll go ahead and merge it into the rest of the branches.

If you have used the beta before then be sure to pull the latest copy of the beta image with docker image pull jacobalberty/unifi:beta before testing.

[jamesmr89]

Ah I saw that in the startup script but wasn't sure. I need to run on the lts branch right now, is it possible to spin a build off that branch to make the Xms/Xmx configurable via these environmentals?

Jim

On Wed, May 9, 2018 at 8:37 AM, Jacob Alberty notifications@github.com wrote:

Ok, if you use jacobalberty/unifi:beta (you'll need to pass the right PKGURL for the version you use) you can now use the environment variables JVM_MAX_HEAP_SIZE (Default is 1024M) to set -Xmx and JVM_INIT_HEAP_SIZE (no default) to set -Xms .

If you haven't used the beta tag before then just a note it installs the package at PKGURL at run time instead of at build time so it's a little slower on the first docker up, latest stable is 5.7.23 so to run that docker up -e PKGURL=https://dl.ubnt.com/unifi/5.7.23/unifi_sysvinit_ all.deb jacobalberty/unifi:beta . Add more -e options for JVM_MAX_HEAP_SIZE and JVM_INIT_HEAP_SIZE to suit your needs, if it works then let me know and I'll go ahead and merge it into the rest of the branches.

If you have used the beta before then be sure to pull the latest copy of the beta image with docker image pull jacobalberty/unifi:beta before testing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jacobalberty/unifi-docker/issues/119#issuecomment-387781309, or mute the thread https://github.com/notifications/unsubscribe-auth/AAErl7H6Omhepne5nrCgkCICwuewQd94ks5tww0dgaJpZM4T3v9j .

[jacobalberty]

Is there any reason you can't use the beta tag with the lts version? -e PKGURL=https://dl.ubnt.com/unifi/5.6.37/unifi_sysvinit_all.deb would give you lts on the beta tag.

[jamesmr89]

ah that works.

On Wed, May 9, 2018 at 10:12 AM, Jacob Alberty notifications@github.com wrote:

Is there any reason you can't use the beta tag with the lts version? -e PKGURL=https://dl.ubnt.com/unifi/5.6.37/unifi_sysvinit_all.deb would give you lts on the beta tag.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jacobalberty/unifi-docker/issues/119#issuecomment-387809880, or mute the thread https://github.com/notifications/unsubscribe-auth/AAErl1RvKqKeHj1l7lygJHyUGfr588bVks5twyN3gaJpZM4T3v9j .

[jamesmr89]

java -Dunifi.datadir=/unifi/data -Dunifi.logdir=/unifi/log -Dunifi.rundir=/var/run/unifi -Xmx4096m -Xmx1024M that works fine, thanks lets see how this does now.

Thanks,

On Wed, May 9, 2018 at 10:16 AM, James Records james.records@gmail.com wrote:

ah that works.

On Wed, May 9, 2018 at 10:12 AM, Jacob Alberty notifications@github.com wrote:

Is there any reason you can't use the beta tag with the lts version? -e PKGURL=https://dl.ubnt.com/unifi/5.6.37/unifi_sysvinit_all.deb would give you lts on the beta tag.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jacobalberty/unifi-docker/issues/119#issuecomment-387809880, or mute the thread https://github.com/notifications/unsubscribe-auth/AAErl1RvKqKeHj1l7lygJHyUGfr588bVks5twyN3gaJpZM4T3v9j .

[jamesmr89]

Hmm,

Crashed again last night.

I'm reading this post: https://help.ubnt.com/hc/en-us/articles/115005159588-UniFi-How-to-Tune-the-Controller-for-High-Number-of-UniFi-Devices

is there any way to set this via the beta currently? unifi.G1GC.enabled=true

Also any possibility of getting jstat added?

Currently I have around 70 sites with about 350 devices. a simple restart of the container seems to fix most issues relating to this, but attempting to get it more stable.

There is also this:

https://community.ubnt.com/t5/UniFi-Wireless/Controller-appears-to-be-hitting-load-limit-what-to-do/td-p/2270477

[jacobalberty]

jstat requires using openjdk-8-jdk instead of -jre which adds around 40 mb more to the image, i don't see an effective way to make swapping between the two easier to save that space for production but i'll investigate restructuring to support that, could be useful . adding unifi.G1GC.enable should be fairly easy if you edit /unifi/data/system.properties you can add it there. I'm hesitant to just add a new flag to the environment variables without confirming it solves the problem but maybe if we can find a decent combination that works I can add an environment variable to enable the settings for lots of devices.

I am looking and it looks like I accidentally duplicated the -Xmx parameter. perhaps the second one was overriding the first. I just rebuilt it on the hub with corrections for the -Xmx pameter.

I'm also looking at adding a flag to set sensible defaults for a high number of devices

[jamesmr89]

I'm still having to reboot about once a day on average. I'm using 12G for the xmx setting. I'm wondering if using oracle java would have any benefits?

re: https://community.ubnt.com/t5/UniFi-Wireless/UniFi-Controller-Crashing-All-the-Time/td-p/2056828

[recklessop]

is this still an active issue that the settings aren't being passed from the system.properties to the jvm ?

[varac]

I had the same issue (multiple java.lang.OutOfMemoryError: GC overhead limit exceeded errors in the log) and can confirm that setting the JVM_MAX_HEAP_SIZE environment var to 2048M fixed it for me (as documented in https://hub.docker.com/r/jacobalberty/unifi).

[github-actions[bot]]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.