Internal MQTT broker causing memory leak in docker image

[squirtbrnr]

Make sure you are running the latest version of Home Assistant before reporting an issue.

You should only file an issue if you found a bug. Feature and enhancement requests should go in the Feature Requests section of our community forum:

Home Assistant release (hass --version): 0.60.1 (Docker)

Python release (python3 --version): 3.6.3

Component/platform: Docker image

Description of problem: After a few days the memory consumption of the docker container will balloon and start to cause performance issues in HA. After just 48 hours of up time, the memory used has doubled from ~120MB to ~280MB. Closer to 7-8 days of uptime and the memory consumption can be as high as ~600MB. At this point MQTT and Z-Wave start exhibiting slow downs and decreased performance and many commands sent over either will take a long time to send/receive or ultimately fail.

Expected: Memory consumption should not balloon as uptime increases. Small increases are acceptable, but increasing by ~100MB per day seems excessive.

Problem-relevant configuration.yaml entries and steps to reproduce:

Traceback (if applicable):

Additional info: See this thread for a few screenshots and a couple users reporting similar behavior (myself included): https://community.home-assistant.io/t/memory-leak-hass-docker-container-version/25213

[pvizeli]

Do you use cameras with SSL? There is a bug inside aiohttp.

[ciotlosm]

I see the same problem and I don't use cameras with SSL.

• I'm using hass.io
• I use nginx with certbot and SSL in nginx not in homeassistant
• I have history and recorder disabled, everything is pushed to influxdb

Notes:

• I will have to check next time when memory up if actually it's container restart or host reboot that cleans up the memory.
• I'm running on RPI3

Below my graph on longer period, with memory drops due to host restart:

[squirtbrnr]

I do have two cameras. One generic mjpeg, the other using the Android IP Webcam component. I also have SSL configured through Let's Encrypt on my Synology Diskstation.

[squirtbrnr]

I have a feeling the memory leak might be linked to the Wunderground platform. Wunderground has been having some issues as of late. Not just the component, but also the developer api provided from Wunderground. There are tons of errors in the HA log and the sensors are even showing as "Unknown" or taking a long time to populate. I restarted the Docker container and not even 8 hours later the memory usage went from 120MB to almost 400MB.

[squirtbrnr]

this is very frustrating and I'd be happy to provide logs if needed. I have disabled the Wunderground component and the memory usage still increases over time. Currently I have to restart the docker container every 5 or so days, otherwise MQTT and Z-Wave just stops working intermittently.

[jjlawren]

I’m tracking down a similar memory leak, but i’m running on Raspbian with Python 3.6.3.

I also use MQTT (but with Mosquito) and a zwave stick. I do not use wunderground or have any cameras.

[squirtbrnr]

I guess I forgot to update here. About a week ago I switched over to an external MQTT Broker. I spun up a Mosquitto Docker Container and re-pointed my HA install to it. Since then, I have had minimal memory usage creep. after 6+ days it will get upwards of 300MB but not nearly the 890MB I once saw it at. Occasionally the system will spike in memory usage but it will recover without intervention. Additionally, since switching to an external MQTT broker, the performance of MQTT has improved in my system.

[squirtbrnr]

internal MQTT broker has a memory leak. Since moving to an external MQTT broker I have not experienced this issue.

[fabaff]

I assume that you are referring to https://github.com/beerfactory/hbmqtt/issues/62.

[bvansambeek]

@fabaff , no...(answering in doubt)... I was referring to this issue. Not sure how the beerfactory is related? Or do I miss something?

[fabaff]

That's the broker we are using if a user want to use with the embedded MQTT broker.

[balloobbot]

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates.

Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment :+1:

[bvansambeek]

This issue is related to the HBMQTT broker issue https://github.com/beerfactory/hbmqtt/issues/62 which is used for the internal broker. On github the issue #62 is still open, but not sure if it is fixed yet, as I moved to the hassio add-on. I asked at that issue thread.

If (the issue still exists), then {place a warning under the internal broker with a link to the issue of HBMQTT} AND {close this issue}? placed warning pull request: https://github.com/home-assistant/home-assistant.io/pull/5905

[squirtbrnr]

As this issue is not HA specific, rather an issue with a component it uses which is maintained by someone else, this issue can be closed. All we can do in HA is have a note warning of a possible memory leak and reference the issue with the component. As stated above, I’m using an external broker and have not had the issue since.