search

hassio-addons / addon-unifi

UniFi Network Application - Home Assistant Community Add-ons
https://addons.community
MIT License
278 stars 138 forks source link

Memory Leak #317

Closed Kev1000000 closed 2 years ago

Kev1000000 commented 2 years ago

Problem/Motivation

The memory footprint of this add on continues to grow to a point where it eventually crashes. It starts out at ~800MB of RAM and slowly climbs to > 2GB, and then crashes. I've increased the max memory to 4GB, and that just elongates the eventual crash.

Expected behavior

For the RAM use to stay at reasonable levels over time.

Actual behavior

Continues to grow over time until it eventually runs out of memory and crashes.

Steps to reproduce

Simply wait a few days or weeks for the max memory limit to be hit and then crash.

Note, I do use the Unifi integration to track devices and use those within HA to track person status. Not sure if that is related. I have about 70 clients within my Unifi instance, but I only am tracking devices on 2 SSIDs, totalling around 20 devices.

brendantay commented 2 years ago

I believe I am experiencing the same issue. Have just had to restart due to apparently running out of memory. I do not use the integration though. I will confirm if this happens again and check memory stats too.

arjanhs commented 2 years ago

Having the same issue, Unifi crashes on daily base.

Thrasher2020 commented 2 years ago

Same issue here. I have set up an automation to restart the Unifi add-on when my memory hits 80% and then it drops back to 40% and is ok for 12-18 hours before it happens again.

jaruba commented 2 years ago

same here, started happening when i moved HA from an RPI to an Intel NUC (although it may just as well be from a Unifi plugin update..)

crashes happen anywhere between 1 - 3 days, but usually within 1 day.

Ubuntu 20.04.3 LTS Docker v20.10.11 HA Supervisor v2021.10.8 HA Core v2021.11.5 Unifi Plugin v1.1.1

(i should note that it had this same issue with many HA Core / Supervisor and Unify Plugin versions in the past too, updated many times and no changes)

my settings:

my logs:

[11:00:03] INFO: Now starting the UniFi Network Application...
[11:00:03] INFO: Note: No add-on specific logs beyond this point.
Exception in thread "https-jsse-nio-8443-Acceptor-0" Exception in thread "cron4j::scheduler[a3233a0243c201425cb321400000017d8ef8d00c4cc87f74]::timer[a3233a0243c201425cb321400000017d8ef8d00d0e471891]" Exception in thread "shutdown-listener" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb" Exception in thread "webapi-90" Exception in thread "webapi-86" Exception in thread "webapi-88" java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.tomcat.util.net.SecureNioChannel.<init>(SecureNioChannel.java:80)
    at org.apache.tomcat.util.net.NioEndpoint.setSocketOptions(NioEndpoint.java:415)
    at org.apache.tomcat.util.net.NioEndpoint$Acceptor.run(NioEndpoint.java:508)
    at java.lang.Thread.run(Thread.java:748)
java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-238" Exception in thread "inform-239" java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-224" Exception in thread "inform-205" Exception in thread "inform-236" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-237" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "inform-234"
Exception in thread "inform-234" Exception in thread "cron4j::scheduler[a3233a0243c201425cb321400000017d8ef8d00f715041de]::launcher[a3233a0243c201420bbc20840000017d96a074452d97cde2]" java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-73" Exception in thread "inform-232" java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-97" java.lang.OutOfMemoryError: Java heap space
Exception in thread "WsSitesList" java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-229" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-0" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "discover" Exception in thread "inform-242" java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-91" java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-93" Exception in thread "inform-231" Exception in thread "inform-215" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" Exception in thread "pool-5-thread-1" Exception in thread "inform-241" Exception in thread "inform-222" Exception in thread "https-jsse-nio-8443-ClientPoller-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-3-thread-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-89" Exception in thread "webapi-92" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "stat-processor" java.lang.OutOfMemoryError: Java heap space
Exception in thread "https-jsse-nio-8443-AsyncTimeout" java.lang.OutOfMemoryError: Java heap space
Exception in thread "https-jsse-nio-8443-ClientPoller-0" Exception in thread "https-jsse-nio-8843-ClientPoller-0" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-235" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-2-thread-4" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb" java.lang.OutOfMemoryError: Java heap space
Exception in thread "StunServerListener" Exception in thread "http-nio-8880-ClientPoller-1" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8880-ClientPoller-0"
java.lang.OutOfMemoryError: Java heap space
Exception in thread "https-jsse-nio-8843-ClientPoller-1" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "cron4j::scheduler[a3233a0243c201425cb321400000017d8ef8d00f715041de]::timer[a3233a0243c201425cb321400000017d8ef8d00f377da2c2]"
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-5-thread-2" Exception in thread "http-nio-8080-ClientPoller-1" Exception in thread "pool-3-thread-2" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-4-thread-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-3-thread-3" Exception in thread "pool-4-thread-2" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8080-ClientPoller-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-3-thread-4" java.lang.OutOfMemoryError: Java heap space
cappadanna commented 2 years ago

I have the same problem. Happens every other day.

Exception in thread "stat-processor" java.lang.OutOfMemoryError: Java heap space at java.lang.StringCoding.decode(StringCoding.java:215) at java.lang.String.(String.java:463) at java.lang.String.(String.java:515) at org.bson.io.ByteBufferBsonInput.readString(ByteBufferBsonInput.java:160) at org.bson.io.ByteBufferBsonInput.readCString(ByteBufferBsonInput.java:139) at org.bson.BsonBinaryReader.readBsonType(BsonBinaryReader.java:122) at com.mongodb.DBObjectCodec.readDocument(DBObjectCodec.java:345) at com.mongodb.DBObjectCodec.decode(DBObjectCodec.java:138) at com.mongodb.DBObjectCodec.decode(DBObjectCodec.java:61) at com.mongodb.CompoundDBObjectCodec.decode(CompoundDBObjectCodec.java:43) at com.mongodb.CompoundDBObjectCodec.decode(CompoundDBObjectCodec.java:27) at com.mongodb.operation.CommandResultArrayCodec.decode(CommandResultArrayCodec.java:52) at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:53) at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) at org.bson.codecs.configuration.LazyCodec.decode(LazyCodec.java:47) at org.bson.codecs.BsonDocumentCodec.readValue(BsonDocumentCodec.java:101) at com.mongodb.operation.CommandResultDocumentCodec.readValue(CommandResultDocumentCodec.java:56) at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:84) at org.bson.codecs.BsonDocumentCodec.decode(BsonDocumentCodec.java:41) at com.mongodb.connection.ReplyMessage.(ReplyMessage.java:57) at com.mongodb.connection.CommandProtocol.getResponseDocument(CommandProtocol.java:139) at com.mongodb.connection.CommandProtocol.execute(CommandProtocol.java:118) at com.mongodb.connection.DefaultServer$DefaultServerProtocolExecutor.execute(DefaultServer.java:168) at com.mongodb.connection.DefaultServerConnection.executeProtocol(DefaultServerConnection.java:289) at com.mongodb.connection.DefaultServerConnection.command(DefaultServerConnection.java:176) at com.mongodb.operation.QueryBatchCursor.getMore(QueryBatchCursor.java:207) at com.mongodb.operation.QueryBatchCursor.hasNext(QueryBatchCursor.java:103) at com.mongodb.MongoBatchCursorAdapter.hasNext(MongoBatchCursorAdapter.java:46) at com.mongodb.DBCursor.hasNext(DBCursor.java:145) at com.ubnt.service.system.null.Ò00000(Unknown Source) at com.ubnt.service.system.j.Ò00000(Unknown Source) Exception in thread "http-nio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space Exception in thread "discover" java.lang.OutOfMemoryError: Java heap space Exception in thread "RemoteLoggerServer" java.lang.OutOfMemoryError: Java heap space Exception in thread "inform-315" Exception in thread "WsSitesList" java.lang.OutOfMemoryError: Java heap space java.lang.NullPointerException Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "MQTT Rec: 34e9da44-1b4f-4cbb-b399-a9c36d33e7b4" Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "pool-5-thread-1" at com.ubnt.service.sdn.V$_o.run(Unknown Source) at java.util.TimerThread.mainLoop(Timer.java:555) at java.util.TimerThread.run(Timer.java:505) Exception in thread "MQTT Con: 34e9da44-1b4f-4cbb-b399-a9c36d33e7b4" java.lang.OutOfMemoryError: Java heap space Exception in thread "inform_stat-1" java.lang.OutOfMemoryError: Java heap space

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!

jaruba commented 2 years ago

this should not be marked as stale, the issue persists

signaleye commented 2 years ago

Having the same issue here.... integrations dont get info from the controller cause controller crashed...trying to reach controller...get a white blank page...have to restart the Addon...even watchdog dont help..

jaruba commented 2 years ago

i even have a (scary) visual representation of this memory leak.. this is on a NUC10I5FNH2 with a Intel Core i5-10210U Processor (6M Cache, up to 4.20 GHz), and this integration eats it all up very fast

Screenshot_20211224-234245_2

And this is what happens when I force stop the Unifi integration (before it just dies itself)

PXL_20211224_214358369

jaruba commented 2 years ago

I bought a Home Assistant Blue, migrated my whole HA setup to it, this integration (latest version) on an official supported device, nothing else installed except HA, and it still has the memory leak and crashes constantly.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 00-banner.sh: executing... 
-----------------------------------------------------------
 Add-on: UniFi Network Application
 Manage your UniFi network using a web browser
-----------------------------------------------------------
 Add-on version: 1.1.4
 You are running the latest version of this add-on.
 System: Home Assistant OS 7.4  (aarch64 / odroid-n2)
 Home Assistant Core: 2022.2.7
 Home Assistant Supervisor: 2022.01.1
-----------------------------------------------------------
 Please, share the above information when looking for help
 or support in, e.g., GitHub, forums or the Discord chat.
-----------------------------------------------------------
[cont-init.d] 00-banner.sh: exited 0.
[cont-init.d] 01-log-level.sh: executing... 
[cont-init.d] 01-log-level.sh: exited 0.
[cont-init.d] unifi.sh: executing... 
[cont-init.d] unifi.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[01:29:35] INFO: Now starting the UniFi Network Application...
[01:29:35] INFO: Note: No add-on specific logs beyond this point.
Exception in thread "http-nio-8080-Acceptor-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "inform-85" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "inform-85"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "https-jsse-nio-8443-AsyncTimeout"
Exception in thread "cron4j::scheduler[10f97846003909780561c0310000017f04defef328954826]::timer[10f97846003909780561c0310000017f04defef3562d6a8a]" Exception in thread "webapi-9" java.lang.OutOfMemoryError: Java heap space
Exception in thread "Timer-0" Exception in thread "http-nio-8880-ClientPoller-1" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
Exception in thread "inform-79" Exception in thread "inform-84" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "https-jsse-nio-8443-Acceptor-0"
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "http-nio-8880-ClientPoller-1"
Exception in thread "https-jsse-nio-8843-ClientPoller-1" Exception in thread "pool-5-thread-1" java.lang.OutOfMemoryError: Java heap space
Exception in thread "cluster-ClusterId{value='620d88f2e9d29101478226e6', description='null'}-localhost:27117" Exception in thread "cron4j::scheduler[10f97846003909780561c0310000017f04defee506ae419a]::timer[10f97846003909780561c0310000017f04defee85cb0fc54]" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-10" Exception in thread "inform-83" Exception in thread "webapi-11" 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "webapi-11"
java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "ContainerBackgroundProcessor[StandardEngine[Tomcat]]" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "webapi-7" java.lang.OutOfMemoryError: Java heap space
Exception in thread "WsSitesList" java.lang.NullPointerException
    at com.ubnt.service.sdn.V$_o.run(Unknown Source)
    at java.util.TimerThread.mainLoop(Timer.java:555)
    at java.util.TimerThread.run(Timer.java:505)
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb:651174838" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8880-ClientPoller-0" Exception in thread "https-jsse-nio-8443-ClientPoller-0" java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
Exception in thread "shutdown-listener" java.lang.OutOfMemoryError: Java heap space
Exception in thread "stat-processor" java.lang.OutOfMemoryError: Java heap space
Exception in thread "discover" java.lang.OutOfMemoryError: Java heap space
Exception in thread "MQTT Con: 8e122033-8f0e-4711-8197-8b23e21aa8fb:651174838" java.lang.OutOfMemoryError: Java heap space
Exception in thread "http-nio-8080-ClientPoller-0" java.lang.OutOfMemoryError: Java heap space
Exception in thread "pool-2-thread-3" java.lang.OutOfMemoryError: Java heap space
signaleye commented 2 years ago

Just an update kinda solve it

I just put the Memory alocation for the container to 2Gb anto all was resolved. Unifi controller is memory hungry.. when I saw the logs it was asking for more tuan 1Gb as you installed it by default in the addons

just change the memroy to whatever tou want and it will not crash

jaruba commented 2 years ago

@signaleye

kinda solve it

are you sure? can you confirm it's still solved?

I just put the Memory alocation for the container to 2Gb anto all was resolved. Unifi controller is memory hungry.. when I saw the logs it was asking for more tuan 1Gb as you installed it by default in the addons

so you increased the RAM memory cap for the docker container to 2gb instead of 1gb from the terminal? I'd need to try to root the HA Blue device to do that, as it doesn't allow me to make changes to docker containers of official integrations (they don't even show up in portainer)

TheEffbee commented 2 years ago

I think he means in the Configuration section in the add-on, it has a place to set minimum and maximum memory.

jaruba commented 2 years ago

@TheEffbee @signaleye

If that's what he meant, then it does not fix this issue, as I did state in a previous comment:

my settings:

Initial Memory: 256 Max Memory: 1024 (also tried with 2048, but it still crashes)

TheEffbee commented 2 years ago

So far on my Raspberry Pi4 4GB it has been running for two days with no crashes, after changing:

Initial Mem: 1024 Max Mem: 2048

Add-on RAM Usage has been between 23-26%

jaruba commented 2 years ago

@TheEffbee i think it will still crash for u too later on

I tried with init memory 512 and max memory 2048 on the Home Assistant Blue device after my last comment and it died in less than 10h, I also tried these settings on a powerful intel nuc and it had no effect there either, max time the addon didn't crash for me was about 7 days (i think)

TheEffbee commented 2 years ago

@jaruba It's been around two weeks and no crashes yet. Mem Usage at 30.8%

jaruba commented 2 years ago

@TheEffbee thanks for reporting back, 2 days after ur message i did an update to both the OS and HA, it didn't go down since then for me either, i'm scared to jinx it though.. but it did go down after trying to increase the mem limit (numerous times) so my hopes are that the updates helped, maybe the mem leak was in the java version used 😕

I'll ping back if it dies out again

jaruba commented 2 years ago

I can confirm that this has been running fine for me too for a long time now, it seems that the OS and HA update fixed it for me

github-actions[bot] commented 2 years ago

There hasn't been any activity on this issue recently, so we clean up some of the older and inactive issues. Please make sure to update to the latest version and check if that solves the issue. Let us know if that works for you by leaving a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thanks!