[Bug] Memory Leak?

[debsahu]

Describe the bug Cant pinpoint the exact version when this started, but this addon has been taking over 8GB of RAM memory

To Reproduce Steps to reproduce the behavior:

1. Disable addon and see RAM usage stop ballooning. (11PM-11AM)
2. Enable the addon and the RAM skyrockets up to 8GB or more. (before 11PM & after 11AM)

Expected behavior RAM usage shouldn't explode when the addon is used.

System configuration System: Supervised HASSIO HA Version: 2021.7.4 WyzeApi Version: 2021.7.11

home-assistant.log

SnakeViz report over 60s

Nothing out of ordinary, no errors. Just lot of calls to Wyze servers. Intentionally blurred out access tokens to 'XXXXXXXXXXXXXXXXXXXXXXX'
---------------------------
.
.
.
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] Request:
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] url: https://api.wyzecam.com/app/v2/device/get_event_list
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] json: {'phone_id': 'eef6fb82-5253-452f-b300-46a21fcaeac0', 'begin_time': 1627413949095, 'event_type': '', 'app_name': 'com.hualai.WyzeCam', 'count': 10, 'app_version': '2.18.43', 'order_by': 2, 'event_value_list': ['1', '13', '10', '12'], 'sc': '9f275790cab94a72bd206c8876429f3c', 'device_mac_list': [], 'event_tag_list': [], 'sv': '782ced6909a44d92a1f70d582bbe88be', 'end_time': 1627417549095, 'phone_system_type': '1', 'app_ver': 'com.hualai.WyzeCam___2.18.43', 'ts': 1623612037763, 'device_mac': '', 'access_token': 'XXXXXXXXXXXXXXXXXXXXXXX'}
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] headers: None
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] data: None
2021-07-27 16:25:49 DEBUG (Thread-19) [wyzeapy.services.sensor_service] Providing update for Kitchen Pantry Motion
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] Request:
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] url: https://api.wyzecam.com/app/v2/device/get_device_Info
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] json: {'phone_system_type': '1', 'app_version': '2.18.43', 'app_ver': 'com.hualai.WyzeCam___2.18.43', 'device_mac': 'YYYYYYYYYY', 'sc': '9f275790cab94a72bd206c8876429f3c', 'ts': 1627417549, 'device_model': 'PIR3U', 'sv': 'c86fa16fc99d4d6580f82ef3b942e586', 'access_token': 'XXXXXXXXXXXXXXXXXXXXXXX', 'phone_id': 'eef6fb82-5253-452f-b300-46a21fcaeac0', 'app_name': 'com.hualai.WyzeCam'}
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] headers: None
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] data: None
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] Request:
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] url: https://api.wyzecam.com/app/v2/device/get_property_list
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] json: {'phone_system_type': '1', 'app_version': '2.18.43', 'app_ver': 'com.hualai.WyzeCam___2.18.43', 'sc': '9f275790cab94a72bd206c8876429f3c', 'ts': 1627417549, 'sv': '9d74946e652647e9b6c9d59326aef104', 'access_token': 'XXXXXXXXXXXXXXXXXXXXXXX', 'phone_id': 'eef6fb82-5253-452f-b300-46a21fcaeac0', 'app_name': 'com.hualai.WyzeCam', 'device_model': 'WYZECP1_JEF', 'device_mac': 'YYYYYYYYYY', 'target_pid_list': []}
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] headers: None
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] data: None
2021-07-27 16:25:49 DEBUG (Thread-19) [wyzeapy.services.sensor_service] Providing update for Fridge
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] Request:
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] url: https://api.wyzecam.com/app/v2/device/get_device_Info
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] json: {'phone_system_type': '1', 'app_version': '2.18.43', 'app_ver': 'com.hualai.WyzeCam___2.18.43', 'device_mac': 'YYYYYYYYYY', 'sc': '9f275790cab94a72bd206c8876429f3c', 'ts': 1627417549, 'device_model': 'DWS3U', 'sv': 'c86fa16fc99d4d6580f82ef3b942e586', 'access_token': 'XXXXXXXXXXXXXXXXXXXXXXX', 'phone_id': 'eef6fb82-5253-452f-b300-46a21fcaeac0', 'app_name': 'com.hualai.WyzeCam'}
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] headers: None
2021-07-27 16:25:49 DEBUG (MainThread) [wyzeapy.wyze_auth_lib] data: None

[SecKatie]

Sorry I cannot replicate. Docker stats is only reporting around 1.7 GB of ram used by my home-assistant container

[SecKatie]

If you can show where a memory leak is than I am happy to merge a pr/create a commit

[tggman]

Ever since the July HAC update, my Home Assistant installation has been crashing (i.e. HA becomes completely unresponsive) once every 24 hours (give or take).

Unfortunately, the only way for me to recover is to power down/up my Home Assistant installation - which of course means I lose my logs. I am however, able to review my memory use history right after a crash/reboot and can clearly see it climb at a constant rate all the way to 100% over that 24 hour period / just prior to the crash.

My next step will be to disable the ha-wyzeapi integration to see if the issue goes away and will report back.

In about the same timeframe, I switched my HA installation over from a Pi4 to an ODRIOD N2+ (AKA Home Assistant Blue) so that may be related and indicate that the issue MIGHT be unique to the ODRIOD N2+? @debsahu what hardware platform is your HA installation running on.

Note: I have15 Wyze bulbs and 4 cameras in my HA installation.

[debsahu]

It might be this addon along with python 3.9 (shipped with 2021.7.4 HA). Experience the same behavior as @tggman, crashes overnight with iowait being way too high and load exceeding 80(not exaggerating here). When I turn on logs for wyzeapi the logs keep rolling like the matrix movie, my GUESS is this addon is making lots of asynchronous cloud polling REST API calls.

Hardware:

I have custom built hardware: Intel i5-10400, Asus B460M-a motherboard with AEGIS 8GB DDR4 RAM (16GB SWAP on SSD).

Software:

Adhered to ADR-0012

System Health

version	core-2021.7.4
installation_type	Home Assistant Supervised
dev	false
hassio	true
docker	true
virtualenv	false
python_version	3.9.5
os_name	Linux
os_version	5.10.0-0.bpo.7-amd64
arch	x86_64
timezone	America/Detroit

Home Assistant Community Store

GitHub API | ok -- | -- Github API Calls Remaining | 4889 Installed Version | 1.13.2 Stage | running Available Repositories | 854 Installed Repositories | 17

Home Assistant Cloud

logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Debian GNU/Linux 10 (buster) -- | -- update_channel | stable supervisor_version | supervisor-2021.06.8 docker_version | 20.10.7 disk_total | 1927.2 GB disk_used | 60.7 GB healthy | true supported | true supervisor_api | ok version_api | ok installed_addons | Home Assistant Google Drive Backup (0.104.3), Samba share (9.5.1), Node-RED (9.2.1), Glances (0.12.3), File editor (5.3.3), WireGuard (0.5.1), Network UPS Tools (0.7.1), FTP (4.2.0), Mosquitto broker (6.0.1), TasmoAdmin (0.15.0), Portainer (1.5.1), Grafana (7.0.4), Terminal & SSH (9.1.3), Visual Studio Code (3.6.0), InfluxDB (4.1.1), motionEye (0.15.0), NGINX Home Assistant SSL proxy (3.0.2), Log Viewer (0.11.1), TasmoBackup (1.04.17), Let's Encrypt (4.11.0), AdGuard Home (4.1.6), Nginx Proxy Manager (0.11.0), eWeLink Smart Home (1.0.0), Frigate NVR (1.13), ESPHome (1.20.1), MariaDB (2.4.0), phpMyAdmin (0.4.3)

Lovelace

dashboards | 1 -- | -- resources | 6 views | 7 mode | storage

[SecKatie]

@debsahu it does make async calls to the cloud (and a lot of them depending on how many sensors you have) which might be why you are seeing this.

[sudodaemon]

I have been having the same issue... After a few days of trouble shooting I was down to two different integrations it could be, and when I disabled this integration, memory use returned to normal.

I had to create an automation to reboot when memory use hit 80% otherwise everything was unusable.

So, now to dig deeper to see whats causing this.

[SecKatie]

I found this on the python bug tracker https://bugs.python.org/issue44697

[SecKatie]

Quick Q for @sudodaemon @debsahu and @tggman: If you downgrade to the previous version of HA does this still occur?

[SecKatie]

https://github.com/home-assistant/core/issues/53530 Are all of you running supervised?

[tggman]

Yes.

Note: As part of my debugging process, I did a partial HAC (ONLY) roll back to v2021.7.2 from v2021.7.4 and YES the issue persisted AFTER the HAC rollback. However, as @debsahu mentioned, python 3.9 was shipped with HAC 2021.7.4 and it was apparently? NOT rolled back when I rolled back HAC to v2021.7.2.

System Health

version	core-2021.7.2
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
virtualenv	false
python_version	3.9.5
os_name	Linux
os_version	5.9.16
arch	aarch64
timezone	America/Chicago

AccuWeather

can_reach_server | ok -- | -- remaining_requests | 43

Home Assistant Cloud

logged_in | true -- | -- subscription_expiration | August 13, 2021, 7:00 PM relayer_connected | true remote_enabled | false remote_connected | false alexa_enabled | true google_enabled | true can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Home Assistant OS 6.1 -- | -- update_channel | stable supervisor_version | supervisor-2021.06.8 docker_version | 20.10.6 disk_total | 113.9 GB disk_used | 7.3 GB healthy | true supported | true board | odroid-n2 supervisor_api | ok version_api | ok installed_addons | Samba share (9.5.1), Terminal & SSH (9.1.3), Let's Encrypt (4.11.0), Duck DNS (1.13.0), Z-Wave JS (0.1.29)

Lovelace

dashboards | 1 -- | -- resources | 0 views | 13 mode | storage

[SecKatie]

@tggman have you tried running v2021.6 like is mentioned in the issue I linked above?

[tggman]

I did not try rolling back to a version earlier then v2021.7.2 since I did not "seem" to have the issue until after I updated to a later version. Note: I can't be 100% certain because I make changes to my yaml code and reboot all the time.

I also rolled back Z-Wave JS from 1.3? to v1.29 when I rolled back HAC to v2021.7.2

So again, could it be the python engine update that triggered/is the problem?

[SecKatie]

Potentially because there was no major change to how the integration performs it's job between now and in the earlier versions. I think this is either a core or a python problem.

IMHO though it would be great if someone could point to a line or lines of code from this project that have a memory leak issue. My suspicion is that there is a memory leak in the asyncio or aiohttp libraries that I use to make calls to the api and the reason why it seems to fix the problem when you disable this integration is that I make so many more web calls than other integrations

[debsahu]

I don’t have a 2021.6.x snapshot to roll back into. I’ll try to get you some info from home assistant “profiler”. There are a bunch of asyncio calls from this addon, might be linked to the Python bug you pointed out.

[debsahu]

kcachegrind for 60s:

SnakeViz for same 60s:

• camera_service.py: line 35 -> base_service.py line 212/96
• sensor_service.py: line 28 -> base_service.py line 451

Heapy Profile Browser for 60s

Object Logs to track heap:

2021-07-28 16:52:24 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('dict', 302083, 302083), ('list', 235627, 235627), ('File', 162284, 162284), ('function', 118215, 118215), ('tuple', 99748, 99748), ('cell', 66140, 66140), ('weakref', 29020, 29020), ('type', 13324, 13324), ('LRItem', 13117, 13117), ('builtin_function_or_method', 12624, 12624), ('getset_descriptor', 11461, 11461), ('method', 9865, 9865), ('set', 9708, 9708), ('property', 8691, 8691), ('ModuleSpec', 5102, 5102), ('module', 5097, 5097), ('SourceFileLoader', 4869, 4869), ('Production', 4342, 4342), ('member_descriptor', 4128, 4128), ('wrapper_descriptor', 3657, 3657), ('method_descriptor', 3603, 3603), ('Schema', 3260, 3260), ('HassJob', 3088, 3088), ('OidOrderedDict', 2704, 2704), ('OrderedDict', 2371, 2371), ('UUID', 2369, 2369), ('Characteristic', 2090, 2090), ('classmethod', 1970, 1970), ('frozenset', 1875, 1875), ('Context', 1780, 1780), ('staticmethod', 1692, 1692), ('mappingproxy', 1684, 1684), ('InstanceState', 1586, 1586), ('PosixPath', 1561, 1561), ('Message', 1496, 1496), ('Optional', 1463, 1463), ('State', 1335, 1335), ('Node', 1236, 1236), ('_abc_data', 1197, 1197), ('PathMetadata', 1154, 1154), ('DistInfoDistribution', 1153, 1153), ('ABCMeta', 1134, 1134), ('uint8_t', 1097, 1097), ('_tuplegetter', 989, 989), ('deque', 953, 953), ('Parameter', 902, 902), ('MibScalarInstance', 897, 897), ('MibIdentifier', 884, 884), ('Counter32', 873, 873), ('RepositoryData', 854, 854), ('RepositoryContent', 854, 854), ('RepositoryInformation', 854, 854), ('RepositoryStatus', 854, 854), ('Validate', 854, 854), ('RepositoryReleases', 854, 854), ('RepositoryVersions', 854, 854), ('RepositoryPath', 854, 854), ('HacsManifest', 854, 854), ('HassLogger', 848, 848), ('_GenericAlias', 827, 827), ('RegistryEntry', 827, 827), ('Service', 821, 821), ('Param', 817, 817), ('Events', 793, 793), ('States', 793, 793), ('MibScalar', 780, 780), ('Required', 765, 765), ('FileFinder', 758, 758), ('_UnionGenericAlias', 750, 750), ('cython_function_or_method', 698, 698), ('AIOGitHubAPIRepositoryTreeContent', 690, 690), ('Command', 682, 682), ('All', 617, 617), ('Field', 607, 607), ('ConstraintsIntersection', 601, 601), ('FieldDescriptor', 591, 591), ('_FieldProperty', 580, 580), ('MibTableColumn', 572, 572), ('_Element', 568, 568), ('itemgetter', 564, 564), ('Attribute', 563, 563), ('uint16_t', 563, 563), ('HacsIntegration', 554, 554), ('Registry', 536, 536), ('Device', 524, 524), ('StructField', 523, 523), ('EnumMeta', 499, 499), ('partial', 499, 499), ('PointerType', 489, 489), ('fused_cython_function', 483, 483), ('ClusterId', 467, 467), ('EntitySubscription', 445, 445), ('weakproxy', 419, 419), ('_Listener', 417, 417), ('CommandHeader', 415, 415), ('Integration', 396, 396), ('CType', 378, 378), ('ForwardRef', 375, 375), ('ValueSizeConstraint', 358, 358), ('MovedAttribute', 346, 346)]
2021-07-28 16:52:55 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('File', 167604, 5320), ('list', 240925, 5298), ('dict', 307362, 5279), ('TimerHandle', 230, 62), ('Context', 1827, 47), ('State', 1346, 11), ('mappingproxy', 1694, 10), ('SimpleCookie', 22, 4), ('CIMultiDict', 75, 3), ('partial', 501, 2), ('Lock', 216, 2), ('Connection', 12, 2), ('ConnectionKey', 13, 2), ('ClientResponse', 4, 2), ('RequestInfo', 4, 2), ('TimeoutHandle', 5, 2), ('ClientRequest', 4, 2), ('coroutine_wrapper', 2, 2), ('JsonPayload', 2, 2), ('MQTTMessage', 1, 1), ('MQTTMessageInfo', 1, 1), ('UUID', 2370, 1), ('WSMessage', 12, 1), ('HassJob', 3089, 1), ('suppress', 7, 1), ('LogEntry', 15, 1), ('DeviceStatus', 44, 1), ('Message', 1497, 1), ('Device', 525, 1), ('APIConnection', 12, 1), ('HTTPMessage', 2, 1), ('AddrInfo', 1, 1), ('IPv4Sockaddr', 1, 1)]
2021-07-28 16:53:25 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('File', 172884, 5280), ('dict', 312632, 5270), ('list', 246191, 5266), ('TimerHandle', 270, 40), ('Context', 1863, 36), ('mappingproxy', 1698, 4), ('State', 1350, 4)]
2021-07-28 16:53:55 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('dict', 317835, 5203), ('list', 251362, 5171), ('File', 178044, 5160), ('Message', 1500, 3), ('WSMessage', 13, 1)]
2021-07-28 16:54:25 CRITICAL (SyncWorker_27) [homeassistant.components.profiler] Memory Growth: [('list', 256683, 5321), ('File', 183364, 5320), ('dict', 323126, 5291), ('HTTPMessage', 3, 1), ('Message', 1501, 1), ('HTTPResponse', 3, 1)]
2021-07-28 16:54:55 CRITICAL (SyncWorker_22) [homeassistant.components.profiler] Memory Growth: [('dict', 328479, 5353), ('list', 262000, 5317), ('File', 188604, 5240), ('Context', 1885, 22), ('TimerHandle', 280, 10), ('Event', 295, 10), ('Handle', 84, 3), ('Message', 1502, 1), ('CIMultiDict', 76, 1), ('ConnectionKey', 14, 1), ('TimerContext', 28, 1), ('SSLProtocol', 21, 1), ('_SSLProtocolTransport', 21, 1), ('_SSLPipe', 21, 1), ('SSLObject', 21, 1), ('ClientConnectionError', 1, 1)]
2021-07-28 16:55:25 CRITICAL (SyncWorker_24) [homeassistant.components.profiler] Memory Growth: [('File', 193524, 4920), ('list', 266838, 4838), ('dict', 333307, 4828), ('Message', 1504, 2), ('TaskStepMethWrapper', 3, 1), ('frozenset', 1876, 1), ('Quoter', 1, 1)]
2021-07-28 16:55:55 CRITICAL (SyncWorker_26) [homeassistant.components.profiler] Memory Growth: [('list', 271925, 5087), ('dict', 338389, 5082), ('File', 198524, 5000), ('Message', 1505, 1), ('DNSAddress', 43, 1)]
2021-07-28 16:56:25 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('File', 203804, 5280), ('list', 277124, 5199), ('dict', 343578, 5189), ('Message', 1511, 6), ('SplitResult', 36, 1)]
2021-07-28 16:56:55 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('dict', 348733, 5155), ('list', 282258, 5134), ('File', 208851, 5047), ('DNSQuestion', 12, 4), ('DNSService', 46, 3), ('StreamReader', 38, 1), ('Message', 1512, 1), ('socket', 105, 1), ('SelectorKey', 76, 1), ('KeyedRef', 175, 1), ('_SelectorSocketTransport', 51, 1), ('TransportSocket', 68, 1), ('ResponseHandler', 28, 1), ('HttpResponseParser', 26, 1), ('TimerContext', 29, 1)]
2021-07-28 16:57:25 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('dict', 353989, 5256), ('File', 214091, 5240), ('list', 287491, 5233), ('Message', 1517, 5), ('WSMessage', 14, 1)]
2021-07-28 16:57:55 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('File', 219411, 5320), ('list', 292797, 5306), ('dict', 359283, 5294), ('Message', 1518, 1)]
2021-07-28 16:58:25 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('dict', 364573, 5290), ('list', 298085, 5288), ('File', 224691, 5280), ('memoryview', 3, 1)]
2021-07-28 16:58:55 CRITICAL (SyncWorker_26) [homeassistant.components.profiler] Memory Growth: [('list', 302902, 4817), ('File', 229491, 4800), ('dict', 369340, 4767), ('Message', 1521, 3), ('Event', 297, 2), ('RawResponseMessage', 1, 1), ('InputContainer', 2, 1), ('Dictionary', 1, 1)]
2021-07-28 16:59:25 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('File', 234153, 4662), ('list', 307486, 4584), ('dict', 373830, 4490), ('Message', 1522, 1)]
2021-07-28 16:59:55 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('dict', 378912, 5082), ('list', 312530, 5044), ('File', 239111, 4958), ('TimerHandle', 286, 6), ('Context', 1891, 6), ('HttpRequestParser', 11, 5), ('socket', 109, 4), ('SelectorKey', 80, 4), ('KeyedRef', 179, 4), ('_SelectorSocketTransport', 55, 4), ('TransportSocket', 72, 4), ('RequestHandler', 11, 3), ('AccessLogger', 11, 3), ('Future', 122, 2), ('deque', 954, 1), ('HassJob', 3090, 1), ('SSLProtocol', 22, 1), ('_SSLProtocolTransport', 22, 1), ('_SSLPipe', 22, 1), ('SSLObject', 22, 1), ('SubscribeMessage', 1, 1), ('PublishMetadata', 1, 1), ('LockOperationActivity', 2, 1), ('TimerNoop', 1, 1)]
2021-07-28 17:00:25 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('File', 243255, 4144), ('list', 316618, 4088), ('dict', 382995, 4083), ('coroutine', 311, 17), ('frame', 316, 17), ('CIMultiDict', 79, 3), ('list_iterator', 8, 2), ('Message', 1524, 2), ('CeilTimeout', 2, 2), ('HTTPSHandler', 2, 1), ('OpenerDirector', 3, 1), ('UnknownHandler', 2, 1), ('HTTPHandler', 2, 1), ('HTTPDefaultErrorHandler', 3, 1), ('HTTPRedirectHandler', 3, 1), ('FTPHandler', 2, 1), ('FileHandler', 3, 1), ('HTTPErrorProcessor', 3, 1), ('DataHandler', 2, 1), ('Future', 123, 1), ('coroutine_wrapper', 3, 1), ('TimeoutHandle', 6, 1), ('SimpleCookie', 23, 1), ('ClientRequest', 5, 1), ('_TransportPlaceholder', 2, 1), ('timeout', 7, 1)]
2021-07-28 17:00:55 CRITICAL (SyncWorker_26) [homeassistant.components.profiler] Memory Growth: [('dict', 387761, 4766), ('File', 247991, 4736), ('list', 321339, 4721), ('builtin_function_or_method', 12627, 3), ('MQTTMessage', 2, 1), ('MQTTMessageInfo', 2, 1), ('Message', 1525, 1), ('_ParserDictionaryContext', 13, 1), ('XMLParser', 13, 1), ('_ErrorLog', 24, 1), ('_TempStore', 24, 1), ('_ParserContext', 23, 1)]
2021-07-28 17:01:25 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('File', 252727, 4736), ('dict', 392492, 4731), ('list', 326067, 4728), ('Message', 1530, 5)]
2021-07-28 17:01:55 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('dict', 397609, 5117), ('list', 331149, 5082), ('File', 257722, 4995), ('DNSService', 48, 2), ('deque', 955, 1), ('Message', 1531, 1), ('TimerContext', 30, 1), ('socket', 110, 1), ('SelectorKey', 81, 1), ('KeyedRef', 180, 1), ('_SelectorSocketTransport', 56, 1), ('TransportSocket', 73, 1)]
2021-07-28 17:02:25 CRITICAL (SyncWorker_23) [homeassistant.components.profiler] Memory Growth: [('File', 262384, 4662), ('list', 335798, 4649), ('dict', 402242, 4633), ('Message', 1532, 1)]

[SecKatie]

@debsahu I am not super familiar with those tools can you help me to understand what I am looking at?

[sudodaemon]

So, after some more troubleshooting, this issue seems to be only with certain entities enabled. I have a ton of Wyze stuff... 8 cameras, lock, doorbell, v2 sensors, I don't use all of them in HA, but the entities are there... So of the 36 entities available to me, I have all but the main 7 I use disabled. All my v2 sense stuff bricked, so all the entries in HA are old and just need to be deleted, etc.... Anyways, I'll dig into this more tomorrow to see if I can isolate the entity(s) that are causing me issues.

[sudodaemon]

Just wanted to give an update of where I'm at. Made it the entire night with the Wyze integration on, with just lights, plugs, and lock, and memory was stable. I then enabled half of the previously disabled entities, boom, memory leak again. So from here I rebooted, and only enabled the binary_sensors... Memory leak is still present

I still need to troubleshoot more and further isolate. But my digging is making me think it's something to do with just binary_sensors...

[sudodaemon]

It's something with binary_sensor. All other entities are fine, the minute I enable any of the binary_sensors associated with various Wyze devices, memory begins to disappear. I'd be interested to know if anybody else experiencing this memory issue has it resolved by disabling all wyze binary_sensors and rebooting. I personally believe based off when this issue started occurring for me, that it was a system update/python update that started this... it was not directly after any Wyze Integration update. I'm at least pleased that I can disable all binary sensors for the time being until a resolution is found.

Once the memory leak starts, it doesn't stop without a reboot. I can boot with no binary sensors, let it run for 30 minutes and see stable memory... the minute I enable a binary sensor, the memory usage increases, and will not stop increasing even if I disable the binary sensor I enabled.

[hellboyfishman]

I don't have much new information, but I'm seeing the same as sudodaemon, as soon as I disabled the binary motion sensors for the wyze cameras the memory seems much more stable. I also don't use those in any automations so it isn't a big deal to me to leave them off for now. I'm going to keep an eye on it but that might be what's causing the python issues.

[SecKatie]

Ok so now that I know what direction to look I can see if there is anything in the code path for sensors that could be causing this.

[SecKatie]

Thank you @sudodaemon for the help in narrowing down where the problem might be

[sudodaemon]

Glad to help! If you need me to test anything just let me know.

[tggman]

I can confirm that disabling the binary sensors associated with my Wyze cams appears to have solved the memory use issue AND works well as a containment action.

Thanks to both @sudodaemon for narrowing it down and to @debsahu for opening up the issue and getting us all heading down the right path!!

@JoshuaMulliken may I suggest you temporarily remove or disable these sensors until a fix is found ... and as always, thanks to you for creating and maintaining this important integration.

[SecKatie]

@tggman I won't be disabling it for now bc it is working in my setup (and likely others) I have not had a chance to dig into the code yet but I expect to this weekend

[tggman]

Working in your setup? That's interesting. I assumed this problem was across the board. Are you running the latest version of HAC and Python? i.e. what makes your setup different from those of us that have the issue? Also, did plan on making the Home Assistant team aware of the memory issue on their end? Specifically. that filling up memory crashes HAC? That should not happen regardless ...

[SecKatie]

@tggman I am running in a container (unsupervised)

System Health

version	core-2021.7.4
installation_type	Home Assistant Container
dev	false
hassio	false
docker	true
virtualenv	false
python_version	3.9.5
os_name	Linux
os_version	5.8.0-63-generic
arch	x86_64
timezone	America/New_York

Home Assistant Community Store

GitHub API | ok -- | -- Github API Calls Remaining | 4920 Installed Version | 1.13.2 Stage | running Available Repositories | 848 Installed Repositories | 3

Home Assistant Cloud

logged_in | true -- | -- subscription_expiration | August 12, 2021, 8:00 PM relayer_connected | true remote_enabled | true remote_connected | false alexa_enabled | true google_enabled | false can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Lovelace

dashboards | 1 -- | -- resources | 0 views | 4 mode | storage

Spotify

api_endpoint_reachable | ok -- | --

[tggman]

Obviously you will let us all know what you find (if anything) once you dig into your code. After that, let me know if you would like me to open up an issue with the Home Assistant team relative to the 100% memory usage = crash - or at the very least, makes it totally unresponsive (HAC / Samba / SSH)

[SecKatie]

For the life of me I cannot find a place where the code would cause any of these problems. It would be great if a python dev could provide a second set of eyes.

[debsahu]

@JoshuaMulliken I don’t know the source of the issue. I provided the info that home assistant devs suggested to provide using “profiler”. I’m no Python developer, but from the looks of it the first two give idea of what functions are called and how much time is spent in each function over 60s. The next two track heap which is useful to troubleshoot what variables are created over certain time.

The good news is my issues seem to have resolved. I used “ha core update --version 2021.6.6” to downgrade. Same issue with memory leak there. Updated the Wyzeapi addon to 2021.7.12. Upgraded supervised HA: “ ha core update --version 2021.7.4”. And no more memory leaks. Can’t narrow down how it was resolved, documenting here so that others can replicate.

[SecKatie]

@JoshuaMulliken I don’t know the source of the issue. I provided the info that home assistant devs suggested to provide using “profiler”. I’m no Python developer, but from the looks of it the first two give idea of what functions are called and how much time is spent in each function over 60s. The next two track heap which is useful to troubleshoot what variables are created over certain time.

The good news is my issues seem to have resolved. I used “ha core update --version 2021.6.6” to downgrade. Same issue with memory leak there. Updated the Wyzeapi addon to 2021.7.12. Upgraded supervised HA: “ ha core update --version 2021.7.4”. And no more memory leaks. Can’t narrow down how it was resolved, documenting here so that others can replicate.

@tggman can you replicate these findings?

[tggman]

@JoshuaMulliken I don’t know the source of the issue. I provided the info that home assistant devs suggested to provide using “profiler”. I’m no Python developer, but from the looks of it the first two give idea of what functions are called and how much time is spent in each function over 60s. The next two track heap which is useful to troubleshoot what variables are created over certain time. The good news is my issues seem to have resolved. I used “ha core update --version 2021.6.6” to downgrade. Same issue with memory leak there. Updated the Wyzeapi addon to 2021.7.12. Upgraded supervised HA: “ ha core update --version 2021.7.4”. And no more memory leaks. Can’t narrow down how it was resolved, documenting here so that others can replicate.

@tggman can you replicate these findings?

I already tried downgrading my HAC installation to a point BEFORE the memory leak became noticeable (i.e. a daily issue). Specifically, I downgraded to HAC 2021.7.2. But downgrading did NOT fix the problem. And since the Python update was pushed through at about the same time and apparently was NOT rolled back when I downgraded to HAC 2021.7.2, that Python update is still suspect and @JoshuaMulliken you will not be able to fix the issue until Python is fixed ??

As it stands now, I've been running HAC without the wyzeapi binary sensors and without a reboot for around 5 days. And while disabling the binary sensors eliminated the daily crash, I'm not convinced the problem was 100% resolved by this action. My memory usage continued to grow - all be it at a SIGNIFICANTLY slower rate - from around the typical 30% to 45% over that 5 day period.

Home Assistant OS 6.2 (https://github.com/home-assistant/operating-system/releases) was pushed through today which I just installed and which of course required a reboot. I don't see any mention of a memory leak or python fix but I'll continue to monitor the memory use, and report back after a few days. I also took this opportunity to update wyzeapi from 2021.7.11 to 2021.7.12 (@JoshuaMulliken what changed?)

For the sake of completeness, here is my current Home Assistant configuration as of today.

System Health

version	core-2021.7.4
installation_type	Home Assistant OS
dev	false
hassio	true
docker	true
virtualenv	false
python_version	3.9.5
os_name	Linux
os_version	5.10.53
arch	aarch64
timezone	America/Chicago

AccuWeather

can_reach_server | ok -- | -- remaining_requests | 33

Home Assistant Cloud

logged_in | true -- | -- subscription_expiration | August 13, 2021, 7:00 PM relayer_connected | true remote_enabled | false remote_connected | false alexa_enabled | true google_enabled | true can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Home Assistant OS 6.2 -- | -- update_channel | stable supervisor_version | supervisor-2021.06.8 docker_version | 20.10.6 disk_total | 113.9 GB disk_used | 7.1 GB healthy | true supported | true board | odroid-n2 supervisor_api | ok version_api | ok installed_addons | Samba share (9.5.1), Terminal & SSH (9.1.3), Let's Encrypt (4.11.0), Duck DNS (1.13.0), Z-Wave JS (0.1.34)

Lovelace

dashboards | 1 -- | -- resources | 0 views | 13 mode | storage

[tggman]

@JoshuaMulliken I don’t know the source of the issue. I provided the info that home assistant devs suggested to provide using “profiler”. I’m no Python developer, but from the looks of it the first two give idea of what functions are called and how much time is spent in each function over 60s. The next two track heap which is useful to troubleshoot what variables are created over certain time.

The good news is my issues seem to have resolved. I used “ha core update --version 2021.6.6” to downgrade. Same issue with memory leak there. Updated the Wyzeapi addon to 2021.7.12. Upgraded supervised HA: “ ha core update --version 2021.7.4”. And no more memory leaks. Can’t narrow down how it was resolved, documenting here so that others can replicate.

@debsahu now that your issue has been resolved, can you please let us know which version of Python you are running? I'm curious to see if it's the version before 3.9.5.

[debsahu]

Python 3.9.5

System Health

version	core-2021.7.4
installation_type	Home Assistant Supervised
dev	false
hassio	true
docker	true
virtualenv	false
python_version	3.9.5
os_name	Linux
os_version	5.10.0-0.bpo.7-amd64
arch	x86_64
timezone	America/Detroit

Home Assistant Community Store

GitHub API | ok -- | -- Github API Calls Remaining | 4892 Installed Version | 1.13.2 Stage | running Available Repositories | 850 Installed Repositories | 17

Home Assistant Cloud

logged_in | false -- | -- can_reach_cert_server | ok can_reach_cloud_auth | ok can_reach_cloud | ok

Home Assistant Supervisor

host_os | Debian GNU/Linux 10 (buster) -- | -- update_channel | stable supervisor_version | supervisor-2021.06.8 docker_version | 20.10.7 disk_total | 1927.2 GB disk_used | 45.1 GB healthy | true supported | true supervisor_api | ok version_api | ok installed_addons | Home Assistant Google Drive Backup (0.104.3), Samba share (9.5.1), Node-RED (9.2.1), Glances (0.12.3), File editor (5.3.3), WireGuard (0.5.1), Network UPS Tools (0.8.0), FTP (4.2.0), Mosquitto broker (6.0.1), TasmoAdmin (0.15.0), Portainer (1.5.1), Grafana (7.0.4), Terminal & SSH (9.1.3), Visual Studio Code (3.6.0), InfluxDB (4.1.1), motionEye (0.15.0), NGINX Home Assistant SSL proxy (3.0.2), Log Viewer (0.11.1), TasmoBackup (1.04.17), Let's Encrypt (4.11.0), AdGuard Home (4.1.6), Nginx Proxy Manager (0.11.0), eWeLink Smart Home (1.0.0), Frigate NVR (1.13), ESPHome (1.20.3), MariaDB (2.4.0), phpMyAdmin (0.4.3), Nextcloud Backup (0.16.2)

Lovelace

dashboards | 1 -- | -- resources | 6 views | 7 mode | storage

[isuelt]

Updated the Wyzeapi addon to 2021.7.12. Upgraded supervised HA: “ ha core update --version 2021.7.4”. And no more memory leaks. Can’t narrow down how it was resolved, documenting here so that others can replicate.

Same here. This resolved the problem for me too.

[debsahu]

@JoshuaMulliken close? Seems to be resolved, don't know how.

[SecKatie]

@debsahu haha I guess!

[tggman]

Okay I just ran: ha core update --version 2021.6.6 to downgrade, and ran: ha core update --version 2021.7.4 to get back where I was. I'll report back as soon as I know if this process fixed the issue.

[debsahu]

Okay I just ran: ha core update --version 2021.6.6 to downgrade, and ran:

I updated this addon in between.

ha core update --version 2021.7.4 to get back where I was. I'll report back as soon as I know if this process fixed the issue.

[tggman]

10 hours later and so far so good. Not knowing root cause is never preferred (problem might come back) and it makes no sense, but I'll take it !!

Here are the (3) things I did different this time vs last time I downgraded: 1) I used an SSH command line to downgrade/restore (ha core update --version 2021.6.6 // ha core update --version 2021.7.4) instead of restoring from an snapshot. 2) wyzeapi was already at 2021.7.12 (vs 2021.7.11) before and after the process 3) The first time I downgraded to HAC 2021.7.2 instead of 2021.6.6, but keep in mind I didn't notice the issue at HAC 2021.7.2

I'll put my money on #1 as the "solution" but who knows. I'll report back IF the problem appears to comes back. Thanks again boys and girls.

[tggman]

I know this was closed (somewhat prematurely), but for the sake of completeness, it looks like using a the above mentioned command line commands fixed my issue.