Memory Leak

[jplandry908]

I am noticing memory leaks on Home Assistant server whenever I have the Blue Iris add-on enabled. If it is enabled, the system memory slowly drops (currently have 6GB allocated) from having 5GB free to 0 and crashing the app. However, if I disable the module the system memory stays flat at 5GB. Is anyone else experiencing a similar issue? I am currently using the latest version of the module v1.0.15 and the latest version of Home Assistant v2022.12.1.

[kramttocs]

I haven't seen this. Running 16 cameras. How long does it take for this to happen? When did it start? What changed around that time?

Do you have debug logging turned on or is it set to Default in the Configure options? Do you have an always on dashboard that shows the streaming view? What if you disable all the motion entities through the Configure menu, or all the audio entities, and so on. Or if you disable all the cameras and just the Server itself is left?

I don't know of a way to debug memory issues without replication so I can only offers suggestions like this on things to try to narrow it down. How you are running HA may have a big thing to do with it also. Happy to help however I can though.

[kramttocs]

Have you checked the forum? There may be ideas to help track this in your environment. Just as an example: https://community.home-assistant.io/t/ha-pi4-4gb-dying-all-ram-used-up/414191/3 Do you have automations or other entities using the Blue Iris entities?

[jplandry908]

Hi @kramttocs. Thank you for your feedback.

I was using "HA v2022.11.5", "BlueIris NVR v1.0.14", and "HACS v1.28.3". All 3 of these were prompting for an upgrade. After upgrading all 3, I started to notice HA was unresponsive at times throughout the day (which I had never experienced before) but would auto-recover on its own. No other changes were made except for the upgrades.

I'm using the Hyper-V (ova) version of HA, so I logged into the VM and noticed an out of memory error. The VM is setup for 6GB and I had rarely see it using more than 2GB previously. I then setup systemmonitor in the config to monitor memory, and could see memory start dropping steadily (about 50m every 5 minutes) and starting to drop after it was running for about 10-30 minutes (it varied in my tests). I also tried disabling audio, motion, and alerts leaving on the camera entities enabled but could still see the memory leak. The only way to stop the memory leak is to disable the BlueIris add-on all together.

As for BlueIris, I'm running 6 cameras. I have area cards showing 4 of the cameras for those zones (not live), but the tabs they browse to have the live video for those zones within them. These are not constantly being displayed since HA is mainly used for automation and accessed via mobile phones. On a side note, another issue I have experienced is that cameras lose their location (after restarts, upgrades, etc) so they are often not shown on the area cards until they are manually placed back into their proper locations.

Since this is a VM with daily backups, I'm going to restore my backup from a few days ago before the upgrades were completed and install systemmonitor to get a baseline for a couple of hours. After that, I will start upgrading the components one at a time monitoring the memory usage after each upgrade and report back my findings.

[jplandry908]

I was able to update HAC (to v1.28.4) and BlueIris (to v1.0.15) without any issues while still running HA v2022.11.5. However, once I upgraded HA to v2022.12.1, the memory leak started to occur. However, if I disable the BlueIris add-on, there is no memory leak. So the issue seems to be linked to HA v2022.12.1 and the BlueIris add-on.

@kramttocs - Have you already upgraded HA to v2022.12.1?

Here is a screenshot of my memory. You can see it running stable for the first hour. This is when HAC, BlueIris, etc were all up-to-date while running HA v2022.11.5. I then upgraded HA to v2022.12.1 and even rebooted the VM. You can immediately see the memory leak in the graph. I then disabled the BlueIris plugin and restarted HA, where you can see the memory leak no longer occurs.

[kramttocs]

Yeah, I am on the latest. I would submit a bug to the HA Core code. Nothing has changed in quite some time in the custom component that would impact this. Things like the semi-newly added services aren't always running. I will close this based on your finding that the HA upgrade is the key factor (they all played together prior to that) but we can reopen if new information warrants it. Thanks.

[jplandry908]

FYI - It appears others are also experiencing memory issues on the 2022.12, so I assume this is not related to the add-on (even though I am only experiencing this when the add-on is enabled).

https://github.com/home-assistant/core/issues/83661