`ReceiveMessage` Memory Leak since updating to 2024.3.0

alexh3o commented 6 months ago

The problem

Hello,

I have been having a steady home assistant core memory usage increase since the exact time I updated to HA 2024.3. Memory usage increases on ha core only, at a very steady pace of around 100MB/hour. Nothing seems to affect this (notably, it doesn't seem linked to yaml reload like on the other open issue). It's also happening in safe mode. Memory is freed when restarting Home Assistant.

I have spent the last weeks trying to figure it out without success, I haven't made any significant change around the time memory leak appeared, other than HA update. I've also done without success, from the CLI: ha core rebuild, ha supervisor repair, ha core check. They all completed succesfully, but this had no effect.

I'm attaching:

graph of first appearance of the memory leak on ha core
same graph in safe mode
profiler logs ran for 4 hours
profiler's callgrind and cprof files, issued after running safe mode for 4 hours I've tried to run py_spy with no luck. I previsouly had an error message (with panick in it, i didn't mark it down), and now bizarely I can't find the home assistant core pid in top command.

Any help or question is welcome :)

AroundUpdate_HACoreMemoryUsageGraph SafeMode4h_HACOreMemoryUsageGraph SafeMode4hours_CallgrindCprof.zip SafeMode4h_ProfilerLogs.log

What version of Home Assistant Core has the issue?

core-2024.3.1

What was the last working version of Home Assistant Core?

core-2024.2.5

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Nothing that wasn't there before

Additional information

Host was a Raspberry Pi 4 8GB
Then I switched to a mini PC with HAOS in a proxmox VM: same problem

oliverde8 commented 6 months ago

Hello,

I am having the ssame issue since updating. I need to restart my ha instance every day, it ends up becoming very very slow and uses all the cpu writing tons of logs:

And each time the system loads get's crazy

I tried disabling as many extensions as I could. But it only gave it more time before it broke.

home-assistant[bot] commented 6 months ago

Hey there @bdraco, mind taking a look at this issue as it has been labeled with an integration (profiler) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `profiler` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign profiler` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

profiler documentation profiler source _{^{(message by IssueLinks)}}

alexh3o commented 6 months ago

Here's the py_spy error blocking me from geting the py spy report

My HA sometimes has slowness, lagging at moments, while working good otherwise, and it's been a mess with the 2 updates to 2024.3.1 and 2024.3.2,with multiple reboots and supervisor unavailable in HA although HA Observer said everything was fine. So maybe I have a hardware problem or else.

Edit: I switched from raspberry pi 4 8GB to a Beelink Mini S12 Pro, with HAOS restored from backup and running in a proxmox VM,. Slowness and lag in HA is all gone, but I still have the memory leak, so it's apparently not linked to hardware. For py_spy i now get a message that py_spy doesn't support python 3.12

woniol commented 6 months ago

Same problem for me. I run version 2024.3.3. Since 2024.3 i need to restart the raspi almost daily.

Tygozwolle commented 5 months ago

Same issue for me.

bdraco commented 5 months ago

https://www.home-assistant.io/integrations/profiler/#service-profilerstart_log_objects

Please provide a log with the profiler.start_log_objects service run for at least 60 minutes at least every 30 seconds

woniol commented 5 months ago

Where can I find the log file after running profiler.start_log_objects ?

Tygozwolle commented 5 months ago

Here is my log


2024-04-02 10:27:39.661 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('dict', 146399, 590), ('tuple', 141177, 554), ('Element', 5221, 387), ('partial', 6463, 377), ('function', 111699, 202), ('HassJob', 4542, 191), ('cell', 64964, 190), ('list', 69365, 149), ('State', 3314, 100), ('TemplateState', 341, 99), ('ReadOnlyDict', 6659, 92), ('ReferenceType', 29016, 63), ('frame', 9794, 31), ('deque', 2704, 24), ('traceback', 8883, 22), ('TraceElement', 383, 20), ('frozenset', 6889, 14), ('set', 18699, 11), ('ReceiveMessage', 1138, 9), ('Context', 5722, 8), ('Events', 45, 6), ('BluetoothServiceInfoBleak', 35, 6), ('BLEDevice', 37, 6), ('AdvertisementData', 37, 6), ('lock', 161, 4), ('Struct', 912, 4), ('Group', 30, 4), ('LogEntry', 39, 4), ('AutomationTrace', 23, 4), ('Event', 2052, 3), ('OrderedDict', 302, 3), ('InstanceState', 70, 3), ('Timeout', 20, 3), ('hamt_bitmap_node', 490, 3), ('CallKey', 17, 3), ('Handle', 61, 2), ('Condition', 75, 2), ('memoryview', 16, 2), ('Thread', 31, 2), ('CaseInsensitiveDict', 275, 2), ('uint3_t', 147, 2), ('uint1_t', 892, 2), ('Request', 4, 2), ('ZCLHeader', 3, 2), ('FrameControl', 3, 2), ('BufferedReader', 5, 1), ('RLock', 67, 1), ('managedbuffer', 11, 1), ('WSMessage', 9, 1), ('_TrackTimeInterval', 91, 1), ('Template', 422, 1), ('SocketIO', 3, 1), ('uint32_t', 362, 1), ('TXStatus', 20, 1), ('DefaultCookiePolicy', 9, 1), ('Empty', 25, 1), ('RequestsCookieJar', 8, 1), ('PreparedRequest', 2, 1), ('StateAttributes', 5, 1), ('Url', 1, 1), ('TypeError', 1, 1), ('EventData', 2, 1), ('_WorkItem', 2, 1), ('_GatheringFuture', 1, 1), ('Read_Attributes', 2, 1), ('AnonymousList', 2, 1), ('StopReason', 3, 1), ('DeliveryError', 1, 1), ('CancelledError', 1, 1), ('SelectedEntities', 1, 1), ('off', 1, 1)]
2024-04-02 10:28:10.076 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('dict', 147319, 920), ('function', 112520, 821), ('cell', 65648, 684), ('tuple', 141799, 622), ('ReadOnlyDict', 7012, 353), ('Context', 5979, 257), ('Element', 5473, 252), ('State', 3558, 244), ('TemplateState', 558, 217), ('list', 69475, 110), ('frozenset', 6949, 60), ('Event', 2099, 47), ('ReferenceType', 29052, 36), ('frame', 9818, 24), ('traceback', 8903, 20), ('InstanceState', 87, 17), ('set', 18713, 14), ('Events', 58, 13), ('hamt_bitmap_node', 498, 8), ('States', 58, 6), ('BluetoothServiceInfoBleak', 41, 6), ('BLEDevice', 43, 6), ('AdvertisementData', 43, 6), ('MultiDictProxy', 15, 3), ('StopReason', 5, 2), ('hamt', 483, 2), ('_QueuedScriptRun', 1, 1), ('TaskStepMethWrapper', 1, 1), ('URL', 22, 1), ('SplitResult', 148, 1), ('RawRequestMessage', 15, 1), ('LogEntry', 40, 1), ('CIMultiDictProxy', 143, 1), ('MultiDict', 12, 1), ('Template', 423, 1), ('Script', 148, 1), ('Empty', 26, 1), ('HomeAssistantRequest', 14, 1), ('RequestHandler', 15, 1), ('StreamWriter', 14, 1), ('UrlMappingMatchInfo', 14, 1), ('AccessLogger', 15, 1), ('StreamReader', 6, 1), ('PayloadAccessError', 4, 1), ('TimerNoop', 4, 1), ('CancelledError', 2, 1), ('ConnectionResetError', 1, 1)]
2024-04-02 10:28:39.804 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('dict', 149153, 1834), ('function', 113596, 1076), ('tuple', 142827, 1028), ('cell', 66563, 915), ('Context', 6577, 598), ('ReadOnlyDict', 7409, 397), ('State', 3848, 290), ('TemplateState', 840, 282), ('TimerHandle', 2430, 267), ('list', 69691, 216), ('method', 13050, 199), ('ReferenceType', 29226, 174), ('set', 18876, 163), ('InstanceState', 248, 161), ('States', 159, 101), ('frozenset', 7026, 77), ('Events', 115, 57), ('hamt_bitmap_node', 506, 8), ('Event', 2106, 7), ('CIMultiDict', 160, 7), ('hamt', 490, 7), ('BluetoothServiceInfoBleak', 47, 6), ('BLEDevice', 49, 6), ('AdvertisementData', 49, 6), ('ReceiveMessage', 1143, 5), ('partial', 6467, 4), ('URL', 26, 4), ('SplitResult', 152, 4), ('RawRequestMessage', 19, 4), ('CIMultiDictProxy', 147, 4), ('HomeAssistantRequest', 18, 4), ('RequestHandler', 19, 4), ('StreamWriter', 18, 4), ('UrlMappingMatchInfo', 18, 4), ('AccessLogger', 19, 4), ('MemoryBIO', 20, 4), ('StreamReader', 9, 3), ('MultiDictProxy', 17, 2), ('_SSLProtocolTransport', 10, 2), ('ActiveConnection', 12, 2), ('WebSocketAdapter', 12, 2), ('_SSLSocket', 17, 2), ('SSLProtocol', 10, 2), ('SSLObject', 10, 2), ('PayloadAccessError', 6, 2), ('TimerNoop', 6, 2), ('managedbuffer', 12, 1), ('WSMessage', 10, 1), ('Task', 67, 1), ('_TrackPointUTCTime', 117, 1), ('KeyedRef', 181, 1), ('SimpleCookie', 13, 1), ('socket', 70, 1), ('LogEntry', 41, 1), ('TransportSocket', 39, 1), ('MultiDict', 13, 1), ('Template', 424, 1), ('_SelectorSocketTransport', 28, 1), ('WebSocketWriter', 7, 1), ('AuthPhase', 7, 1), ('ZLibCompressor', 5, 1), ('WebSocketHandler', 7, 1), ('WebSocketResponse', 7, 1), ('FlowControlDataQueue', 7, 1), ('SelectorKey', 56, 1), ('AutomationTrace', 24, 1), ('ResponseHandler', 3, 1), ('StopReason', 6, 1), ('ConnectionKey', 3, 1), ('HttpResponseParser', 3, 1), ('TimerContext', 3, 1), ('DeconzAddressEndpoint', 1, 1)]
2024-04-02 10:29:09.773 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 115028, 1432), ('tuple', 144136, 1309), ('cell', 67803, 1240), ('dict', 150275, 1122), ('ReadOnlyDict', 7956, 547), ('Context', 7043, 466), ('TemplateState', 1197, 357), ('State', 4203, 355), ('list', 69950, 259), ('frozenset', 7128, 102), ('TimerHandle', 2527, 97), ('HassJob', 4574, 32), ('partial', 6498, 31), ('_TrackPointUTCTime', 138, 21), ('PosixPath', 1577, 12), ('lock', 167, 6), ('BluetoothServiceInfoBleak', 52, 5), ('BLEDevice', 54, 5), ('AdvertisementData', 54, 5), ('Struct', 916, 4), ('Event', 2109, 3), ('Thread', 34, 3), ('traceback', 8906, 3), ('MultiDictProxy', 20, 3), ('hamt_bitmap_node', 509, 3), ('Empty', 29, 3), ('Condition', 77, 2), ('MultiDict', 15, 2), ('coroutine_wrapper', 2, 2), ('_UnionGenericAlias', 1596, 1), ('URL', 27, 1), ('SplitResult', 153, 1), ('RawRequestMessage', 20, 1), ('Task', 68, 1), ('CIMultiDictProxy', 148, 1), ('HomeAssistantRequest', 19, 1), ('RequestHandler', 20, 1), ('StreamWriter', 19, 1), ('UrlMappingMatchInfo', 19, 1), ('AccessLogger', 20, 1), ('APIConnection', 6, 1), ('PayloadAccessError', 7, 1), ('TimerNoop', 7, 1), ('AutomationTrace', 25, 1), ('StopReason', 7, 1), ('_Interrupt', 1, 1), ('__pyx_scope_struct_6_start_connection', 1, 1), ('__pyx_scope_struct_5__do_connect', 1, 1), ('__pyx_scope_struct_1__connect_socket_connect', 1, 1), ('AddrInfo', 1, 1), ('IPv4Sockaddr', 1, 1)]
2024-04-02 10:29:39.800 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('function', 117188, 2160), ('tuple', 146044, 1908), ('cell', 69657, 1854), ('dict', 152081, 1806), ('ReadOnlyDict', 8333, 377), ('list', 70281, 331), ('frozenset', 7282, 154), ('State', 4331, 128), ('Context', 7163, 120), ('TemplateState', 1313, 116), ('method', 13159, 109), ('ReferenceType', 29320, 94), ('InstanceState', 290, 42), ('set', 18917, 41), ('States', 186, 27), ('partial', 6523, 25), ('Events', 133, 18), ('Event', 2125, 16), ('HassJob', 4588, 14), ('BluetoothServiceInfoBleak', 58, 6), ('BLEDevice', 60, 6), ('AdvertisementData', 60, 6), ('Task', 73, 5), ('ReceiveMessage', 1148, 5), ('_TrackPointUTCTime', 142, 4), ('Handle', 64, 3), ('Struct', 918, 2), ('_GeneratorContextManager', 14, 2), ('generator', 20, 2), ('Future', 527, 2), ('hamt', 492, 2), ('int8s', 18, 2), ('TaskStepMethWrapper', 2, 1), ('WSMessage', 11, 1), ('KeyedRef', 182, 1), ('socket', 71, 1), ('TransportSocket', 40, 1), ('ZoneStatus', 12, 1), ('_SelectorSocketTransport', 29, 1), ('SelectorKey', 57, 1), ('AutomationTrace', 26, 1), ('CharacterString', 2, 1), ('ResponseHandler', 4, 1), ('_ScriptRun', 3, 1), ('bitmap8', 1, 1), ('ConnectionKey', 4, 1), ('HttpResponseParser', 4, 1), ('TimerContext', 4, 1)]
2024-04-02 10:30:09.917 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('function', 119390, 2202), ('tuple', 147997, 1953), ('cell', 71551, 1894), ('dict', 152716, 635), ('list', 70749, 468), ('frozenset', 7440, 158), ('deque', 2760, 56), ('Handle', 88, 24), ('KeyedRef', 205, 23), ('socket', 94, 23), ('TransportSocket', 63, 23), ('_SelectorSocketTransport', 52, 23), ('SelectorKey', 80, 23), ('ResponseHandler', 27, 23), ('HttpResponseParser', 27, 23), ('TimerContext', 27, 23), ('CIMultiDict', 182, 22), ('StreamReader', 31, 22), ('AdvertisementData', 66, 6), ('BluetoothServiceInfoBleak', 64, 6), ('BLEDevice', 66, 6), ('AutomationTrace', 28, 2), ('HassJob', 4590, 2), ('_TrackPointUTCTime', 144, 2), ('LimitedSizeDict', 11, 2), ('__CDataFromBuf', 1, 1), ('CaseInsensitiveDict', 276, 1)]
2024-04-02 10:30:39.904 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 120636, 1246), ('dict', 153851, 1135), ('cell', 72640, 1089), ('tuple', 149085, 1088), ('list', 70884, 135), ('frozenset', 7530, 90), ('coroutine', 667, 34), ('frame', 9848, 30), ('partial', 6551, 28), ('builtin_function_or_method', 9764, 20), ('traceback', 8925, 19), ('HassJob', 4604, 14), ('Task', 86, 13), ('State', 4343, 12), ('generator', 29, 9), ('_GeneratorContextManager', 22, 8), ('Future', 535, 8), ('BluetoothServiceInfoBleak', 70, 6), ('BLEDevice', 72, 6), ('AdvertisementData', 72, 6), ('ReceiveMessage', 1153, 5), ('enumerate', 6, 4), ('list_iterator', 10, 4), ('hamt', 496, 4), ('hamt_bitmap_node', 512, 3), ('FutureIter', 41, 3), ('tuple_iterator', 3, 2), ('ScalarInvalid', 2, 2), ('ServiceCall', 3, 2), ('ActiveConnection', 14, 2), ('WebSocketAdapter', 14, 2), ('_AsyncGeneratorContextManager', 4, 1), ('async_generator', 4, 1), ('Schema', 4860, 1), ('MultipleInvalid', 1, 1), ('Condition', 78, 1), ('dict_items', 3, 1), ('chain', 1, 1), ('Lock', 536, 1), ('_TrackPointUTCTime', 145, 1), ('LogEntry', 42, 1), ('AutomationTrace', 29, 1), ('_ScriptRun', 4, 1), ('CancelledError', 3, 1)]
2024-04-02 10:31:09.970 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('function', 122470, 1834), ('tuple', 150663, 1578), ('cell', 74213, 1573), ('dict', 154809, 958), ('list', 71161, 277), ('frozenset', 7661, 131), ('ReadOnlyDict', 8366, 33), ('BluetoothServiceInfoBleak', 77, 7), ('BLEDevice', 79, 7), ('AdvertisementData', 79, 7), ('hamt_bitmap_node', 518, 6), ('State', 4347, 4), ('TraceElement', 386, 3), ('CaseInsensitiveDict', 278, 2), ('generator', 31, 2), ('Future', 537, 2), ('HTTPHeaderDict', 2, 1), ('OrderedDict', 303, 1), ('HTTPResponse', 7, 1), ('Response', 5, 1), ('RequestsCookieJar', 9, 1), ('DefaultCookiePolicy', 10, 1), ('RLock', 68, 1), ('memoryview', 17, 1), ('HassJob', 4605, 1), ('CancelledError', 4, 1), ('_QueuedScriptRun', 2, 1)]
2024-04-02 10:31:39.874 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 125161, 2691), ('tuple', 152973, 2310), ('cell', 76505, 2292), ('dict', 156434, 1625), ('list', 71576, 415), ('frozenset', 7856, 195), ('ReferenceType', 29432, 112), ('method', 13204, 45), ('set', 18945, 28), ('InstanceState', 315, 25), ('Event', 2141, 16), ('States', 197, 11), ('Events', 144, 11), ('URL', 36, 9), ('SplitResult', 162, 9), ('ReadOnlyDict', 8374, 8), ('partial', 6557, 6), ('BluetoothServiceInfoBleak', 83, 6), ('BLEDevice', 85, 6), ('AdvertisementData', 85, 6), ('ReceiveMessage', 1158, 5), ('RawRequestMessage', 24, 4), ('CIMultiDictProxy', 152, 4), ('HomeAssistantRequest', 23, 4), ('RequestHandler', 24, 4), ('StreamWriter', 23, 4), ('UrlMappingMatchInfo', 23, 4), ('AccessLogger', 24, 4), ('hamt', 499, 3), ('BindParameter', 306, 2), ('Lock', 538, 2), ('HassJob', 4607, 2), ('MultiDictProxy', 22, 2), ('ZLibCompressor', 7, 2), ('int16s', 5, 2), ('defaultdict', 979, 1), ('LogEntry', 43, 1), ('TimerNoop', 8, 1), ('int8s', 19, 1), ('ClientConnectionError', 1, 1)]
2024-04-02 10:32:09.932 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 128149, 2988), ('tuple', 155543, 2570), ('cell', 79075, 2570), ('dict', 157289, 855), ('list', 72029, 453), ('frozenset', 8067, 211), ('BluetoothServiceInfoBleak', 89, 6), ('BLEDevice', 91, 6), ('AdvertisementData', 91, 6), ('TraceElement', 392, 6), ('partial', 6559, 2), ('HassJob', 4609, 2)]
2024-04-02 10:32:40.011 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('function', 130263, 2114), ('tuple', 157365, 1822), ('cell', 80867, 1792), ('dict', 158656, 1367), ('list', 72327, 298), ('frozenset', 8219, 152), ('TaskStepMethWrapper', 12, 10), ('BluetoothServiceInfoBleak', 95, 6), ('BLEDevice', 97, 6), ('AdvertisementData', 97, 6), ('ReceiveMessage', 1163, 5), ('MemoryBIO', 24, 4), ('HttpRequestParser', 9, 2), ('SSLProtocol', 12, 2), ('managedbuffer', 14, 2), ('SSLObject', 12, 2), ('memoryview', 18, 1), ('_SSLSocket', 18, 1), ('_SSLProtocolTransport', 11, 1)]
2024-04-02 10:33:10.092 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('function', 132998, 2735), ('tuple', 159783, 2418), ('cell', 83247, 2380), ('dict', 160231, 1575), ('list', 72825, 498), ('frozenset', 8432, 213), ('coroutine', 682, 15), ('BindParameter', 319, 13), ('partial', 6569, 10), ('coroutine_wrapper', 10, 8), ('BinaryExpression', 180, 7), ('Select', 108, 7), ('BluetoothServiceInfoBleak', 101, 6), ('BLEDevice', 103, 6), ('AdvertisementData', 103, 6), ('Event', 2146, 5), ('ResultColumnsEntry', 145, 5), ('_ColumnsPlusNames', 125, 5), ('_ORMColumnEntity', 42, 5), ('AnalyzedFunction', 24, 4), ('LambdaOptions', 24, 4), ('APIConnection', 10, 4), ('_Interrupt', 5, 4), ('__pyx_scope_struct_6_start_connection', 5, 4), ('__pyx_scope_struct_5__do_connect', 5, 4), ('__pyx_scope_struct_1__connect_socket_connect', 5, 4), ('AddrInfo', 5, 4), ('IPv4Sockaddr', 5, 4), ('Timeout', 23, 3), ('LinkedLambdaElement', 6, 3), ('Future', 539, 2), ('_anonymous_label', 100, 2), ('prefix_anon_map', 65, 2), ('PyWrapper', 63, 2), ('list_iterator', 11, 1), ('MySQLCompiler_mysqldb', 43, 1), ('CacheKey', 43, 1), ('CursorResultMetaData', 37, 1), ('ORMSelectCompileState', 26, 1), ('FromLinter', 28, 1), ('default_compile_options', 41, 1), ('StatementLambdaElement', 18, 1)]
2024-04-02 10:33:40.010 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 137821, 4823), ('tuple', 163891, 4108), ('cell', 87349, 4102), ('dict', 162441, 2210), ('list', 73500, 675), ('frozenset', 8777, 345), ('BluetoothServiceInfoBleak', 107, 6), ('BLEDevice', 109, 6), ('AdvertisementData', 109, 6), ('ReceiveMessage', 1168, 5), ('HassJob', 4611, 2), ('int8s', 20, 1), ('int16s', 6, 1)]
2024-04-02 10:34:10.103 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('function', 141236, 3415), ('cell', 90270, 2921), ('tuple', 166757, 2866), ('dict', 164247, 1806), ('list', 74000, 500), ('frozenset', 9021, 244), ('Context', 7238, 75), ('TimerHandle', 2583, 56), ('BluetoothServiceInfoBleak', 113, 6), ('BLEDevice', 115, 6), ('AdvertisementData', 115, 6), ('ZeroconfIPv6Address', 23, 1)]
2024-04-02 10:34:40.178 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 145492, 4256), ('cell', 93967, 3697), ('tuple', 170445, 3688), ('dict', 166466, 2219), ('list', 74672, 672), ('frozenset', 9323, 302), ('hamt_bitmap_node', 540, 22), ('_GeneratorContextManager', 34, 12), ('generator', 40, 9), ('hamt', 507, 8), ('Task', 92, 6), ('BluetoothServiceInfoBleak', 119, 6), ('BLEDevice', 121, 6), ('AdvertisementData', 121, 6), ('Event', 2151, 5), ('_ScriptRun', 8, 4), ('enumerate', 10, 4), ('FutureIter', 44, 3), ('StopReason', 10, 3), ('_AsyncGeneratorContextManager', 7, 3), ('async_generator', 7, 3), ('list_iterator', 13, 2), ('CaseInsensitiveDict', 280, 2), ('ReceiveMessage', 1170, 2), ('PayloadAccessError', 9, 2), ('ScriptRunResult', 2, 2), ('TimerNoop', 9, 1)]
2024-04-02 10:35:10.223 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('function', 150400, 4908), ('tuple', 174841, 4396), ('cell', 98137, 4170), ('dict', 170186, 3720), ('list', 75483, 811), ('Context', 7794, 556), ('State', 4853, 506), ('TemplateState', 1818, 505), ('ReadOnlyDict', 8813, 439), ('frozenset', 9675, 352), ('CaseInsensitiveDict', 322, 42), ('partial', 6599, 30), ('HassJob', 4628, 17), ('Handle', 95, 7), ('deque', 2767, 7), ('BluetoothServiceInfoBleak', 125, 6), ('BLEDevice', 127, 6), ('AdvertisementData', 127, 6), ('CIMultiDict', 187, 5), ('uint8_t', 8289, 3), ('StreamReader', 33, 2), ('int16s', 8, 2), ('ScriptRunResult', 3, 1), ('int8s', 21, 1)]
2024-04-02 10:35:40.381 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('function', 155353, 4953), ('cell', 102468, 4331), ('tuple', 179132, 4291), ('dict', 172389, 2203), ('list', 76204, 721), ('frozenset', 10025, 350), ('coroutine', 756, 74), ('builtin_function_or_method', 9826, 62), ('Task', 121, 29), ('Future', 566, 27), ('FutureIter', 70, 26), ('generator', 62, 22), ('_GeneratorContextManager', 52, 18), ('hamt_bitmap_node', 556, 16), ('hamt', 515, 8), ('Event', 2157, 6), ('list_iterator', 19, 6), ('BluetoothServiceInfoBleak', 131, 6), ('BLEDevice', 133, 6), ('AdvertisementData', 133, 6), ('enumerate', 15, 5), ('_ScriptRun', 11, 3), ('StopReason', 13, 3), ('_AsyncGeneratorContextManager', 10, 3), ('async_generator', 10, 3), ('RLock', 70, 2), ('CaseInsensitiveDict', 324, 2), ('memoryview', 20, 2), ('Condition', 80, 2), ('RequestHandler', 26, 2), ('AccessLogger', 26, 2), ('managedbuffer', 15, 1), ('URL', 37, 1), ('SplitResult', 163, 1), ('RawRequestMessage', 25, 1), ('CIMultiDictProxy', 153, 1), ('Variant', 91, 1), ('uint8_t', 8290, 1), ('HomeAssistantRequest', 24, 1), ('StreamWriter', 24, 1), ('UrlMappingMatchInfo', 24, 1), ('_SSLProtocolTransport', 12, 1), ('_SSLSocket', 19, 1), ('PayloadAccessError', 10, 1), ('TimerNoop', 10, 1), ('AutomationTrace', 30, 1), ('ServiceCall', 4, 1), ('_QueuedScriptRun', 3, 1)]
2024-04-02 10:36:10.504 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 160246, 4893), ('tuple', 183852, 4720), ('cell', 106681, 4213), ('dict', 175350, 2961), ('ReferenceType', 30221, 789), ('list', 76890, 686), ('frozenset', 10374, 349), ('States', 532, 335), ('method', 13480, 276), ('set', 19208, 263), ('InstanceState', 545, 230), ('coroutine', 779, 23), ('Event', 2175, 18), ('FutureIter', 80, 10), ('Future', 575, 9), ('Task', 129, 8), ('BluetoothServiceInfoBleak', 139, 8), ('BLEDevice', 141, 8), ('AdvertisementData', 141, 8), ('_GeneratorContextManager', 56, 4), ('hamt_bitmap_node', 560, 4), ('builtin_function_or_method', 9829, 3), ('uint16_t', 3028, 3), ('partial', 6601, 2), ('Struct', 920, 2), ('HassJob', 4630, 2), ('AutomationTrace', 32, 2), ('_ScriptRun', 13, 2), ('Group', 31, 1), ('list_iterator', 20, 1), ('Command', 7, 1), ('hamt', 516, 1), ('uint8_t', 8291, 1), ('OnOffControl', 2, 1), ('DeviceState', 28, 1), ('LimitedSizeDict', 12, 1), ('int8s', 22, 1), ('ServiceCall', 5, 1), ('Remote', 1, 1), ('RemoteLegacy', 1, 1), ('enumerate', 16, 1), ('_AsyncGeneratorContextManager', 11, 1), ('async_generator', 11, 1), ('_QueuedScriptRun', 4, 1), ('set_iterator', 1, 1), ('Bytes', 1, 1), ('WrappedContextManager', 1, 1)]
2024-04-02 10:36:40.757 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('tuple', 193420, 9568), ('function', 165162, 4916), ('cell', 110847, 4166), ('dict', 179325, 3975), ('list', 79384, 2494), ('set', 20614, 1406), ('History', 893, 892), ('SaveUpdateState', 550, 550), ('ProcessState', 503, 503), ('frozenset', 10728, 354), ('InstanceState', 865, 320), ('method', 13775, 295), ('Events', 338, 194), ('States', 551, 19), ('Preprocess', 8, 8), ('BluetoothServiceInfoBleak', 145, 6), ('BLEDevice', 147, 6), ('AdvertisementData', 147, 6), ('ProcessAll', 6, 6), ('defaultdict', 983, 4), ('uint8_t', 8295, 4), ('SaveUpdateAll', 4, 4), ('DeleteAll', 4, 4), ('PopulateDict', 41, 2), ('immutabledict', 376, 1), ('portable_instancemethod', 78, 1), ('Insert', 25, 1), ('CacheKey', 44, 1), ('ZeroconfIPv6Address', 24, 1), ('SessionTransaction', 2, 1), ('UOWTransaction', 1, 1), ('dict_itemiterator', 1, 1), ('groupby', 1, 1), ('MySQLExecutionContext_mysqldb', 1, 1), ('Cursor', 1, 1), ('_InsertManyValuesBatch', 1, 1)]
2024-04-02 10:37:10.892 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 170051, 4889), ('tuple', 197927, 4507), ('cell', 115054, 4207), ('dict', 181799, 2474), ('list', 79966, 582), ('frozenset', 11077, 349), ('Context', 7915, 121), ('TimerHandle', 2619, 36), ('BluetoothServiceInfoBleak', 152, 7), ('BLEDevice', 154, 7), ('AdvertisementData', 154, 7), ('hamt_bitmap_node', 565, 5), ('Events', 341, 3), ('MultiDictProxy', 24, 2), ('hamt', 518, 2), ('FullyBufferedCursorFetchStrategy', 1, 1), ('CursorResult', 1, 1), ('CursorResultMetaData', 38, 1), ('partial', 6602, 1), ('slice', 30, 1), ('itemgetter', 793, 1), ('URL', 38, 1), ('SplitResult', 164, 1), ('RawRequestMessage', 26, 1), ('CIMultiDictProxy', 154, 1), ('HomeAssistantRequest', 25, 1), ('StreamWriter', 25, 1), ('UrlMappingMatchInfo', 25, 1), ('StopReason', 14, 1)]
2024-04-02 10:37:41.149 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 174976, 4925), ('cell', 119283, 4229), ('tuple', 202048, 4121), ('dict', 184209, 2410), ('list', 80383, 417), ('frozenset', 11428, 351), ('CaseInsensitiveDict', 334, 10), ('BluetoothServiceInfoBleak', 158, 6), ('BLEDevice', 160, 6), ('AdvertisementData', 160, 6), ('partial', 6607, 5), ('HassJob', 4633, 3), ('result', 1, 1), ('int16s', 9, 1)]
2024-04-02 10:38:10.983 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('function', 179854, 4878), ('cell', 123424, 4141), ('tuple', 205786, 3738), ('dict', 186613, 2404), ('list', 80894, 511), ('frozenset', 11779, 351), ('BluetoothServiceInfoBleak', 164, 6), ('BLEDevice', 166, 6), ('AdvertisementData', 166, 6), ('Insert', 26, 1), ('portable_instancemethod', 79, 1), ('PopulateDict', 42, 1), ('CacheKey', 45, 1)]
2024-04-02 10:38:41.479 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 184692, 4838), ('cell', 127558, 4134), ('tuple', 209728, 3942), ('dict', 189034, 2421), ('list', 81293, 399), ('frozenset', 12124, 345), ('BluetoothServiceInfoBleak', 170, 6), ('BLEDevice', 172, 6), ('AdvertisementData', 172, 6)]
2024-04-02 10:39:11.679 CRITICAL (SyncWorker_13) [homeassistant.components.profiler] Memory Growth: [('function', 189454, 4762), ('cell', 131663, 4105), ('tuple', 213477, 3749), ('dict', 191459, 2425), ('frozenset', 12466, 342), ('list', 81489, 196), ('BluetoothServiceInfoBleak', 176, 6), ('BLEDevice', 178, 6), ('AdvertisementData', 178, 6), ('RenderInfo', 25, 1), ('Connection', 13, 1), ('TimeoutHandle', 1, 1), ('ClientRequest', 1, 1), ('HassClientResponse', 1, 1), ('RequestInfo', 1, 1)]
2024-04-02 10:39:41.913 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 194241, 4787), ('tuple', 218180, 4703), ('cell', 135725, 4062), ('dict', 194089, 2630), ('list', 81844, 355), ('frozenset', 12807, 341), ('BluetoothServiceInfoBleak', 183, 7), ('BLEDevice', 185, 7), ('AdvertisementData', 185, 7)]
2024-04-02 10:40:13.037 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 197136, 2895), ('cell', 138221, 2496), ('dict', 194812, 723), ('list', 82370, 526), ('frozenset', 13012, 205), ('deque', 2787, 20), ('State', 4861, 8), ('BluetoothServiceInfoBleak', 190, 7), ('BLEDevice', 192, 7), ('AdvertisementData', 192, 7), ('_TrackPointUTCTime', 146, 1), ('Device', 35, 1), ('TimerContext', 28, 1)]
2024-04-02 10:40:41.252 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 199034, 1898), ('cell', 139821, 1600), ('dict', 195245, 433), ('list', 82751, 381), ('ReadOnlyDict', 9132, 319), ('frozenset', 13149, 137), ('deque', 2831, 44), ('KeyedRef', 217, 12), ('socket', 106, 12), ('TransportSocket', 75, 12), ('_SelectorSocketTransport', 64, 12), ('SelectorKey', 92, 12), ('ResponseHandler', 39, 12), ('HttpResponseParser', 39, 12), ('TimerContext', 39, 11), ('CIMultiDict', 197, 10), ('MemoryBIO', 34, 10), ('StreamReader', 43, 10), ('Handle', 102, 7), ('BluetoothServiceInfoBleak', 196, 6), ('BLEDevice', 198, 6), ('AdvertisementData', 198, 6), ('_SSLProtocolTransport', 17, 5), ('_SSLSocket', 24, 5), ('SSLProtocol', 17, 5), ('SSLObject', 17, 5), ('managedbuffer', 19, 4), ('ConnectionKey', 8, 4), ('memoryview', 23, 3), ('State', 4862, 1), ('HassJob', 4634, 1), ('cycle', 13, 1)]
2024-04-02 10:41:11.948 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 200784, 1750), ('cell', 141316, 1495), ('dict', 196093, 848), ('ReadOnlyDict', 9353, 221), ('frozenset', 13274, 125), ('list', 82860, 109), ('BluetoothServiceInfoBleak', 202, 6), ('BLEDevice', 204, 6), ('AdvertisementData', 204, 6), ('State', 4866, 4), ('ActiveConnection', 15, 1), ('WebSocketAdapter', 15, 1)]
2024-04-02 10:41:42.173 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('function', 202929, 2145), ('cell', 143183, 1867), ('tuple', 219868, 1688), ('dict', 197728, 1635), ('ReadOnlyDict', 9724, 371), ('list', 83170, 310), ('frozenset', 13426, 152), ('BluetoothServiceInfoBleak', 208, 6), ('BLEDevice', 210, 6), ('AdvertisementData', 210, 6), ('State', 4871, 5)]
2024-04-02 10:42:11.828 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 204872, 1943), ('tuple', 221527, 1659), ('cell', 144823, 1640), ('list', 83476, 306), ('dict', 197932, 204), ('frozenset', 13566, 140), ('ReadOnlyDict', 9810, 86), ('BluetoothServiceInfoBleak', 214, 6), ('BLEDevice', 216, 6), ('AdvertisementData', 216, 6), ('EventData', 3, 1)]
2024-04-02 10:42:41.990 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('function', 206889, 2017), ('tuple', 223304, 1777), ('cell', 146597, 1774), ('dict', 199653, 1721), ('list', 83812, 336), ('frozenset', 13709, 143), ('ReadOnlyDict', 9834, 24), ('BluetoothServiceInfoBleak', 220, 6), ('BLEDevice', 222, 6), ('AdvertisementData', 222, 6), ('TraceElement', 393, 1)]
2024-04-02 10:43:11.931 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 208917, 2028), ('cell', 148310, 1713), ('tuple', 225010, 1706), ('dict', 200282, 629), ('list', 84106, 294), ('frozenset', 13854, 145), ('ReadOnlyDict', 9847, 13), ('BluetoothServiceInfoBleak', 226, 6), ('BLEDevice', 228, 6), ('AdvertisementData', 228, 6)]
2024-04-02 10:43:42.068 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('function', 210793, 1876), ('tuple', 226625, 1615), ('cell', 149920, 1610), ('dict', 201214, 932), ('list', 84357, 251), ('frozenset', 13988, 134), ('BluetoothServiceInfoBleak', 232, 6), ('BLEDevice', 234, 6), ('AdvertisementData', 234, 6), ('CaseInsensitiveDict', 335, 1)]
2024-04-02 10:44:12.298 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 212749, 1956), ('tuple', 228312, 1687), ('cell', 151590, 1670), ('dict', 202561, 1347), ('list', 84683, 326), ('frozenset', 14128, 140), ('BluetoothServiceInfoBleak', 238, 6), ('BLEDevice', 240, 6), ('AdvertisementData', 240, 6)]
2024-04-02 10:44:41.956 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 214867, 2118), ('cell', 153412, 1822), ('tuple', 230132, 1820), ('dict', 203605, 1044), ('list', 84985, 302), ('frozenset', 14279, 151), ('BluetoothServiceInfoBleak', 244, 6), ('BLEDevice', 246, 6), ('AdvertisementData', 246, 6)]
2024-04-02 10:45:12.776 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 216672, 1805), ('tuple', 231714, 1582), ('cell', 154955, 1543), ('dict', 204418, 813), ('list', 85267, 282), ('frozenset', 14408, 129), ('CaseInsensitiveDict', 368, 33)]
2024-04-02 10:45:42.202 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('function', 218721, 2049), ('cell', 156719, 1764), ('tuple', 233466, 1752), ('dict', 205437, 1019), ('list', 85526, 259), ('frozenset', 14554, 146), ('TraceElement', 402, 9), ('CaseInsensitiveDict', 370, 2), ('AutomationTrace', 33, 1), ('_WorkItem', 3, 1)]
2024-04-02 10:46:12.032 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 220635, 1914), ('cell', 158356, 1637), ('tuple', 235069, 1603), ('dict', 205750, 313), ('list', 85736, 210), ('frozenset', 14691, 137), ('DNSNsec', 30, 7), ('ZeroconfIPv6Address', 25, 1), ('DNSAddress', 39, 1), ('int8s', 23, 1)]
2024-04-02 10:46:42.157 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('function', 222543, 1908), ('tuple', 236716, 1647), ('cell', 159997, 1641), ('dict', 207349, 1599), ('list', 86063, 327), ('frozenset', 14827, 136), ('TraceElement', 404, 2), ('DNSAddress', 40, 1)]
2024-04-02 10:47:12.294 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 224658, 2115), ('tuple', 238532, 1816), ('cell', 161810, 1813), ('dict', 208144, 795), ('list', 86356, 293), ('frozenset', 14978, 151)]
2024-04-02 10:47:42.176 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('function', 226451, 1793), ('tuple', 240071, 1539), ('cell', 163327, 1517), ('dict', 209356, 1212), ('list', 86613, 257), ('frozenset', 15107, 129), ('int16s', 11, 2), ('int8s', 24, 1), ('TraceElement', 405, 1)]
2024-04-02 10:48:12.333 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 228563, 2112), ('tuple', 241899, 1828), ('cell', 165155, 1828), ('dict', 210421, 1065), ('list', 86935, 322), ('frozenset', 15257, 150), ('State', 4872, 1)]
2024-04-02 10:48:42.105 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('function', 230828, 2265), ('tuple', 243854, 1955), ('cell', 167094, 1939), ('dict', 211630, 1209), ('list', 87319, 384), ('frozenset', 15419, 162), ('TraceElement', 416, 11), ('uint16_t', 3031, 3), ('uint8_t', 8298, 3), ('AutomationTrace', 34, 1)]
2024-04-02 10:49:12.248 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 232438, 1610), ('cell', 168478, 1384), ('tuple', 245233, 1379), ('dict', 212370, 740), ('list', 87530, 211), ('frozenset', 15534, 115)]
2024-04-02 10:49:42.450 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('function', 234678, 2240), ('tuple', 247164, 1931), ('cell', 170394, 1916), ('dict', 213510, 1140), ('list', 87871, 341), ('frozenset', 15694, 160), ('uint8_t', 8300, 2), ('uint16_t', 3032, 1)]
2024-04-02 10:50:12.925 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 236623, 1945), ('tuple', 248842, 1678), ('cell', 172055, 1661), ('dict', 214195, 685), ('list', 88197, 326), ('frozenset', 15833, 139), ('deque', 2838, 7), ('State', 4876, 4), ('_TrackPointUTCTime', 147, 1), ('AutomationTrace', 35, 1), ('LVBytes', 1, 1)]
2024-04-02 10:50:42.409 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 238560, 1937), ('cell', 173600, 1545), ('tuple', 250366, 1524), ('dict', 215243, 1048), ('list', 88385, 188), ('frozenset', 15966, 133), ('getset_descriptor', 11047, 27), ('type', 14842, 21), ('module', 5002, 9), ('ModuleSpec', 4978, 9), ('SourceFileLoader', 4736, 9), ('HassLogger', 948, 4), ('Field', 2573, 3), ('staticmethod', 2664, 2), ('FileFinder', 748, 2), ('_abc_data', 1990, 2), ('_ProtocolMeta', 222, 2), ('_GenericAlias', 2215, 2), ('FlagValue', 24, 2), ('Pattern', 803, 1), ('_DataclassParams', 577, 1), ('PlaceHolder', 157, 1), ('PosixPath', 1578, 1), ('Integration', 171, 1), ('NoPayload', 1, 1)]
2024-04-02 10:51:12.253 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 240781, 2221), ('cell', 175529, 1929), ('tuple', 252284, 1918), ('dict', 216395, 1152), ('list', 88724, 339), ('frozenset', 16124, 158), ('TraceElement', 423, 7), ('OrderedDict', 307, 4), ('Statement', 38, 4), ('Condition', 82, 2), ('Retry', 13, 2), ('CaseInsensitiveDict', 372, 2), ('HTTPAdapter', 12, 2), ('PoolManager', 12, 2), ('RecentlyUsedContainer', 12, 2), ('RLock', 71, 1), ('lock', 168, 1), ('Session', 11, 1), ('finalize', 6, 1), ('_Info', 6, 1), ('LifoQueue', 6, 1), ('PoolKey', 6, 1), ('OpenHardwareMonitorData', 1, 1), ('HTTPConnectionPool', 1, 1), ('HTTPConnection', 1, 1), ('_ZoneTimeoutManager', 1, 1), ('_ZoneTaskContext', 1, 1)]
2024-04-02 10:51:42.076 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('function', 242580, 1799), ('tuple', 253825, 1541), ('cell', 177050, 1521), ('dict', 216943, 548), ('list', 89013, 289), ('frozenset', 16253, 129), ('DNSNsec', 34, 4), ('_ConditionFail', 1, 1)]
2024-04-02 10:52:12.625 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 244692, 2112), ('cell', 178858, 1808), ('tuple', 255630, 1805), ('dict', 218392, 1449), ('list', 89293, 280), ('frozenset', 16401, 148)]
2024-04-02 10:52:42.529 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('function', 246637, 1945), ('tuple', 257301, 1671), ('cell', 180500, 1642), ('dict', 219323, 931), ('list', 89616, 323), ('frozenset', 16541, 140), ('int8s', 25, 1), ('int16s', 12, 1)]
2024-04-02 10:53:12.646 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 248248, 1611), ('tuple', 258735, 1434), ('cell', 181926, 1426), ('dict', 220154, 831), ('list', 89861, 245), ('frozenset', 16655, 114)]
2024-04-02 10:53:42.633 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 250473, 2225), ('tuple', 260625, 1890), ('cell', 183811, 1885), ('dict', 221312, 1158), ('list', 90187, 326), ('frozenset', 16814, 159), ('uint16_t', 3040, 8), ('uint8_t', 8307, 7), ('AddrModeAddress', 99, 5), ('ZigbeePacket', 50, 3), ('SerializableBytes', 50, 3), ('LongOctetString', 49, 3), ('int16s', 14, 2), ('int8s', 26, 1)]
2024-04-02 10:54:12.607 CRITICAL (SyncWorker_13) [homeassistant.components.profiler] Memory Growth: [('function', 252405, 1932), ('tuple', 262272, 1647), ('cell', 185447, 1636), ('dict', 221936, 624), ('list', 90481, 294), ('frozenset', 16953, 139)]
2024-04-02 10:54:42.789 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 254282, 1877), ('tuple', 263922, 1650), ('cell', 187075, 1628), ('dict', 223260, 1324), ('list', 90761, 280), ('frozenset', 17087, 134), ('LogRecord', 2, 1)]
2024-04-02 10:55:13.206 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 256451, 2169), ('tuple', 265768, 1846), ('cell', 188915, 1840), ('dict', 223975, 715), ('list', 91030, 269), ('frozenset', 17242, 155), ('CaseInsensitiveDict', 411, 39), ('ReadOnlyDict', 9870, 23), ('deque', 2840, 2)]
2024-04-02 10:55:42.386 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('function', 258159, 1708), ('cell', 190396, 1481), ('tuple', 267236, 1468), ('dict', 224743, 768), ('list', 91275, 245), ('frozenset', 17371, 129), ('builtin_function_or_method', 9881, 52), ('StateAttributes', 14, 9), ('MemoryBIO', 38, 4), ('memoryview', 25, 2), ('managedbuffer', 21, 2), ('_SSLProtocolTransport', 19, 2), ('_SSLSocket', 26, 2), ('SSLProtocol', 19, 2), ('SSLObject', 19, 2), ('cycle', 14, 1)]
2024-04-02 10:56:12.756 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 260289, 2130), ('cell', 192225, 1829), ('tuple', 269050, 1814), ('dict', 226041, 1298), ('list', 91561, 286), ('frozenset', 17523, 152), ('builtin_function_or_method', 9886, 5), ('uint8_t', 8308, 1)]
2024-04-02 10:56:42.832 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('function', 262207, 1918), ('tuple', 270702, 1652), ('cell', 193867, 1642), ('dict', 227044, 1003), ('list', 91860, 299), ('frozenset', 17660, 137)]
2024-04-02 10:57:12.892 CRITICAL (SyncWorker_13) [homeassistant.components.profiler] Memory Growth: [('function', 263930, 1723), ('cell', 195346, 1479), ('tuple', 272176, 1474), ('dict', 227947, 903), ('list', 92086, 226), ('frozenset', 17783, 123), ('builtin_function_or_method', 9891, 5)]
2024-04-02 10:57:42.794 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('function', 266197, 2267), ('tuple', 274133, 1957), ('cell', 197293, 1947), ('dict', 229066, 1119), ('list', 92463, 377), ('frozenset', 17945, 162), ('uint16_t', 3046, 6), ('AddrModeAddress', 105, 6), ('uint8_t', 8313, 5), ('ZigbeePacket', 53, 3), ('SerializableBytes', 53, 3), ('LongOctetString', 52, 3)]
2024-04-02 10:58:12.786 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 268115, 1918), ('tuple', 275792, 1659), ('cell', 198937, 1644), ('dict', 230104, 1038), ('list', 92719, 256), ('builtin_function_or_method', 10066, 175), ('frozenset', 18082, 137), ('uint16_t', 3050, 4), ('uint8_t', 8317, 4), ('AddrModeAddress', 109, 4), ('NWK', 713, 2), ('ZigbeePacket', 55, 2), ('SerializableBytes', 55, 2), ('LongOctetString', 54, 2), ('State', 4877, 1)]
2024-04-02 10:58:43.104 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 270047, 1932), ('tuple', 277457, 1665), ('cell', 200592, 1655), ('dict', 231041, 937), ('list', 93056, 337), ('frozenset', 18220, 138), ('State', 4885, 8), ('TraceElement', 427, 4), ('uint16_t', 3052, 2), ('uint8_t', 8319, 2), ('NWK', 715, 2), ('AddrModeAddress', 111, 2), ('ZigbeePacket', 56, 1), ('SerializableBytes', 56, 1), ('LongOctetString', 55, 1)]
2024-04-02 10:59:12.872 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 272036, 1989), ('tuple', 279186, 1729), ('cell', 202293, 1701), ('dict', 231989, 948), ('list', 93346, 290), ('frozenset', 18362, 142), ('TraceElement', 497, 70), ('deque', 2846, 6), ('State', 4889, 4), ('CaseInsensitiveDict', 413, 2), ('_TrackPointUTCTime', 149, 2), ('int16s', 15, 1)]
2024-04-02 10:59:41.975 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 273746, 1710), ('cell', 203749, 1456), ('tuple', 280636, 1450), ('dict', 232385, 396), ('list', 93592, 246), ('frozenset', 18485, 123), ('uint16_t', 3058, 6), ('uint8_t', 8325, 6), ('NWK', 721, 6), ('AddrModeAddress', 117, 6), ('ZigbeePacket', 59, 3), ('SerializableBytes', 59, 3), ('LongOctetString', 58, 3)]
2024-04-02 11:00:12.933 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('deque', 2898, 52), ('function', 273777, 31), ('cell', 203774, 25), ('list', 93611, 19), ('tuple', 280655, 19), ('_TrackPointUTCTime', 153, 4), ('frozenset', 18487, 2), ('AutomationTrace', 37, 2)]
2024-04-02 11:00:42.180 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('tuple', 280872, 217), ('function', 273909, 132), ('cell', 203892, 118), ('list', 93699, 88), ('frozenset', 18566, 79), ('BindParameter', 345, 26), ('OrderedSet', 303, 21), ('_anonymous_label', 118, 18), ('_ColumnMetrics', 843, 12), ('ResultColumnsEntry', 157, 12), ('itemgetter', 803, 10), ('Column', 689, 10), ('BinaryExpression', 190, 10), ('_truncated_label', 136, 10), ('_ColumnsPlusNames', 135, 10), ('Select', 116, 8), ('PyWrapper', 69, 6), ('ClauseList', 19, 6), ('_RawColumnEntity', 18, 6), ('prefix_anon_map', 70, 5), ('DDLEventsDispatch', 880, 4), ('Grouping', 20, 4), ('_ORMColumnEntity', 46, 4), ('MySQLCompiler_mysqldb', 46, 3), ('default_compile_options', 44, 3), ('_CompileLabel', 15, 3), ('ColumnSet', 27, 2), ('ReadOnlyColumnCollection', 80, 2), ('_ListenerCollection', 106, 2), ('ColumnCollection', 38, 2), ('ForeignKey', 34, 2), ('Function', 4, 2), ('ORMSelectCompileState', 28, 2), ('FromLinter', 30, 2), ('AnalyzedCode', 19, 2), ('AnalyzedFunction', 26, 2), ('StatementLambdaElement', 20, 2), ('LambdaOptions', 26, 2), ('ColumnClause', 37, 2), ('Subquery', 26, 2), ('attrgetter', 354, 1), ('Comparator', 255, 1), ('Label', 50, 1), ('max', 9, 1), ('min', 2, 1), ('CacheKey', 46, 1), ('InsertDMLState', 16, 1), ('CursorResultMetaData', 39, 1), ('SimpleResultMetaData', 16, 1), ('UnaryExpression', 6, 1), ('_SentinelColumnCharacterization', 6, 1), ('_InsertManyValues', 6, 1), ('Over', 2, 1)]
2024-04-02 11:01:12.092 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('function', 274104, 195), ('cell', 204054, 162), ('tuple', 281029, 157), ('frozenset', 18580, 14)]
2024-04-02 11:01:42.132 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('list', 93745, 46), ('tuple', 281073, 44), ('cell', 204064, 10), ('function', 274109, 5)]
2024-04-02 11:02:12.335 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('dict', 232608, 223), ('function', 274306, 197), ('cell', 204238, 174), ('tuple', 281229, 156), ('list', 93778, 33), ('frozenset', 18594, 14), ('HassJob', 4635, 1)]
2024-04-02 11:02:41.970 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('tuple', 281292, 63), ('cell', 204290, 52), ('function', 274356, 50), ('list', 93790, 12), ('frozenset', 18598, 4)]
2024-04-02 11:03:12.419 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 274372, 16), ('cell', 204291, 1), ('frozenset', 18599, 1)]
2024-04-02 11:03:42.003 CRITICAL (SyncWorker_14) [homeassistant.components.profiler] Memory Growth: [('function', 274594, 222), ('tuple', 281485, 193), ('cell', 204474, 183), ('list', 93860, 70), ('dict', 232667, 59), ('frozenset', 18615, 16)]
2024-04-02 11:04:12.185 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('dict', 232960, 293), ('tuple', 281599, 114), ('cell', 204527, 53), ('function', 274641, 47), ('partial', 6631, 24), ('HassJob', 4657, 22), ('list', 93865, 5), ('frozenset', 18618, 3)]
2024-04-02 11:04:42.189 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('function', 274867, 226), ('dict', 233159, 199), ('cell', 204721, 194), ('tuple', 281730, 131), ('list', 93898, 33), ('frozenset', 18634, 16), ('MQTTMessage', 2, 1), ('MQTTMessageInfo', 2, 1), ('BLEDevice', 247, 1), ('AdvertisementData', 247, 1), ('int8s', 27, 1)]
2024-04-02 11:05:12.724 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('tuple', 281853, 123), ('CaseInsensitiveDict', 450, 37), ('function', 274889, 22), ('cell', 204733, 12), ('deque', 2904, 6), ('frozenset', 18636, 2)]
2024-04-02 11:05:41.986 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 274947, 58), ('cell', 204777, 44), ('frozenset', 18640, 4), ('map', 2, 2), ('AdGuardHome', 3, 1), ('filter', 1, 1), ('chain', 2, 1), ('Prepared', 1, 1), ('suppress', 1, 1), ('TXStatus', 21, 1)]
2024-04-02 11:06:12.217 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 275216, 269), ('cell', 205015, 238), ('tuple', 282051, 198), ('frozenset', 18659, 19), ('list', 93903, 5), ('CaseInsensitiveDict', 452, 2)]
2024-04-02 11:06:41.984 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: []
2024-04-02 11:07:12.386 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('function', 275396, 180), ('cell', 205168, 153), ('tuple', 282193, 142), ('dict', 233253, 94), ('list', 93918, 15), ('frozenset', 18672, 13), ('uint8_t', 8328, 3), ('uint16_t', 3060, 2), ('AddrModeAddress', 119, 2), ('NWK', 722, 1), ('ZigbeePacket', 60, 1), ('SerializableBytes', 60, 1), ('LongOctetString', 59, 1), ('LVBytes', 2, 1)]
2024-04-02 11:07:41.935 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 275490, 94), ('tuple', 282277, 84), ('cell', 205242, 74), ('list', 93945, 27), ('frozenset', 18679, 7)]
2024-04-02 11:08:12.298 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('tuple', 282336, 59), ('list', 93998, 53), ('cell', 205291, 49), ('function', 275537, 47), ('dict', 233277, 24), ('frozenset', 18682, 3), ('uint16_t', 3062, 2), ('uint8_t', 8330, 2), ('NWK', 724, 2), ('AddrModeAddress', 121, 2), ('ZigbeePacket', 61, 1), ('SerializableBytes', 61, 1), ('LongOctetString', 60, 1)]
2024-04-02 11:08:41.981 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 275788, 251), ('tuple', 282551, 215), ('cell', 205504, 213), ('dict', 233473, 196), ('list', 94038, 40), ('frozenset', 18700, 18)]
2024-04-02 11:09:12.158 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('tuple', 282567, 16), ('function', 275798, 10), ('dict', 233481, 8), ('cell', 205510, 6), ('list', 94043, 5), ('_TrackPointUTCTime', 156, 3), ('frozenset', 18701, 1)]
2024-04-02 11:09:42.050 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('dict', 233557, 76), ('tuple', 282629, 62), ('function', 275860, 62), ('cell', 205570, 60), ('list', 94068, 25), ('uint8_t', 8337, 7), ('AddrModeAddress', 127, 6), ('uint16_t', 3067, 5), ('NWK', 729, 5), ('frozenset', 18705, 4), ('ZigbeePacket', 64, 3), ('SerializableBytes', 64, 3), ('LongOctetString', 63, 3)]
2024-04-02 11:10:12.880 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('function', 276038, 178), ('cell', 205719, 149), ('tuple', 282776, 147), ('frozenset', 18718, 13), ('list', 94077, 9), ('deque', 2907, 3), ('PendingRequest', 2, 2), ('AutomationTrace', 38, 1)]
2024-04-02 11:10:42.152 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('function', 276098, 60), ('cell', 205768, 49), ('tuple', 282819, 43), ('SolarlEdgeOptimizer', 20, 10), ('frozenset', 18724, 6), ('NWK', 732, 3), ('uint16_t', 3069, 2), ('AddrModeAddress', 129, 2), ('HTTPBasicAuth', 1, 1), ('HTTPSConnectionPool', 6, 1), ('uint8_t', 8338, 1), ('ZigbeePacket', 65, 1), ('SerializableBytes', 65, 1), ('LongOctetString', 64, 1), ('SolarEdgeSite', 1, 1), ('SolarEdgeInverter', 1, 1), ('SolarEdgeString', 1, 1)]
2024-04-02 11:11:12.259 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 276253, 155), ('cell', 205905, 137), ('tuple', 282931, 112), ('frozenset', 18733, 9), ('int8s', 28, 1)]
2024-04-02 11:11:42.114 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('tuple', 283068, 137), ('function', 276381, 128), ('cell', 206015, 110), ('dict', 233640, 83), ('frozenset', 18742, 9), ('list', 94085, 8), ('LVBytes', 3, 1)]
2024-04-02 11:12:12.372 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 276473, 92), ('cell', 206087, 72), ('dict', 233708, 68), ('tuple', 283120, 52), ('frozenset', 18749, 7), ('list', 94089, 4), ('bitmap32', 1, 1)]
2024-04-02 11:12:42.050 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('dict', 233928, 220), ('function', 276671, 198), ('tuple', 283304, 184), ('cell', 206263, 176), ('list', 94175, 86), ('frozenset', 18763, 14)]
2024-04-02 11:13:12.227 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('tuple', 283387, 83), ('cell', 206333, 70), ('function', 276737, 66), ('list', 94181, 6), ('frozenset', 18768, 5)]
2024-04-02 11:13:41.957 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 276881, 144), ('cell', 206443, 110), ('tuple', 283496, 109), ('list', 94236, 55), ('dict', 233948, 20), ('frozenset', 18778, 10)]
2024-04-02 11:14:12.233 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('function', 277003, 122), ('tuple', 283599, 103), ('cell', 206541, 98), ('dict', 233958, 10), ('frozenset', 18787, 9), ('LVBytes', 4, 1)]
2024-04-02 11:14:42.236 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('dict', 234070, 112), ('tuple', 283700, 101), ('function', 277103, 100), ('cell', 206633, 92), ('list', 94252, 16), ('frozenset', 18794, 7)]
2024-04-02 11:15:12.906 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('tuple', 283873, 173), ('function', 277272, 169), ('cell', 206776, 143), ('list', 94297, 45), ('CaseInsensitiveDict', 489, 37), ('frozenset', 18806, 12), ('DNSAddress', 43, 3), ('DNSNsec', 35, 1)]
2024-04-02 11:15:41.966 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 277384, 112), ('cell', 206872, 96), ('tuple', 283931, 58), ('frozenset', 18814, 8)]
2024-04-02 11:16:12.642 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('tuple', 284041, 110), ('dict', 234165, 95), ('function', 277469, 85), ('cell', 206948, 76), ('frozenset', 18820, 6), ('CaseInsensitiveDict', 491, 2), ('InsecureRequestWarning', 1, 1), ('WarningMessage', 1, 1), ('LogRecord', 3, 1), ('LogEntry', 44, 1)]
2024-04-02 11:16:41.964 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 277650, 181), ('cell', 207100, 152), ('tuple', 284167, 126), ('dict', 234225, 60), ('frozenset', 18833, 13), ('LVBytes', 5, 1)]
2024-04-02 11:17:12.290 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('dict', 234300, 75), ('function', 277702, 52), ('tuple', 284215, 48), ('cell', 207138, 38), ('frozenset', 18837, 4)]
2024-04-02 11:17:42.080 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 277861, 159), ('tuple', 284368, 153), ('cell', 207283, 145), ('dict', 234427, 127), ('list', 94373, 76), ('frozenset', 18848, 11)]
2024-04-02 11:18:12.212 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 277968, 107), ('tuple', 284451, 83), ('cell', 207366, 83), ('dict', 234456, 29), ('frozenset', 18856, 8)]
2024-04-02 11:18:42.232 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('tuple', 284572, 121), ('function', 278084, 116), ('cell', 207472, 106), ('dict', 234520, 64), ('list', 94389, 16), ('frozenset', 18864, 8)]
2024-04-02 11:19:12.337 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('function', 278266, 182), ('cell', 207628, 156), ('tuple', 284727, 155), ('dict', 234648, 128), ('list', 94427, 38), ('frozenset', 18877, 13)]
2024-04-02 11:19:42.174 CRITICAL (SyncWorker_13) [homeassistant.components.profiler] Memory Growth: [('tuple', 284793, 66), ('cell', 207686, 58), ('function', 278324, 58), ('list', 94461, 34), ('dict', 234676, 28), ('frozenset', 18881, 4)]
2024-04-02 11:20:12.885 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('function', 278407, 83), ('tuple', 284867, 74), ('cell', 207755, 69), ('list', 94477, 16), ('frozenset', 18887, 6)]
2024-04-02 11:20:42.245 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 278575, 168), ('cell', 207891, 136), ('tuple', 284981, 114), ('frozenset', 18899, 12)]
2024-04-02 11:21:12.233 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('dict', 234798, 122), ('function', 278672, 97), ('cell', 207976, 85), ('tuple', 285047, 66), ('frozenset', 18906, 7)]
2024-04-02 11:21:42.116 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 278775, 103), ('tuple', 285147, 100), ('cell', 208071, 95), ('dict', 234846, 48), ('frozenset', 18916, 10), ('DNSAddress', 51, 8), ('list', 94483, 6), ('CaseInsensitiveDict', 493, 2)]
2024-04-02 11:22:12.117 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 278864, 89), ('tuple', 285222, 75), ('cell', 208134, 63), ('frozenset', 18920, 4), ('DNSAddress', 52, 1)]
2024-04-02 11:22:42.149 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('dict', 234950, 104), ('tuple', 285302, 80), ('cell', 208202, 68), ('list', 94539, 56), ('function', 278920, 56), ('frozenset', 18924, 4)]
2024-04-02 11:23:12.398 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('function', 279162, 242), ('cell', 208396, 194), ('tuple', 285491, 189), ('dict', 235089, 139), ('list', 94573, 34), ('frozenset', 18941, 17)]
2024-04-02 11:23:42.057 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('list', 94586, 13), ('tuple', 285497, 6), ('dict', 235093, 4)]
2024-04-02 11:24:12.330 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 279326, 164), ('tuple', 285644, 147), ('cell', 208530, 134), ('dict', 235166, 73), ('list', 94604, 18), ('frozenset', 18953, 12)]
2024-04-02 11:24:41.983 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('dict', 235323, 157), ('function', 279467, 141), ('tuple', 285781, 137), ('cell', 208653, 123), ('list', 94652, 48), ('frozenset', 18963, 10), ('BLEDevice', 251, 4), ('AdvertisementData', 251, 4), ('BluetoothServiceInfoBleak', 247, 3)]
2024-04-02 11:25:12.847 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('tuple', 285889, 108), ('cell', 208718, 65), ('function', 279531, 64), ('CaseInsensitiveDict', 524, 31), ('list', 94681, 29), ('frozenset', 18967, 4), ('deque', 2909, 2)]
2024-04-02 11:25:42.408 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('list', 96514, 1833), ('tuple', 286364, 475), ('dict', 235700, 377), ('Name', 326, 306), ('QueryMessage', 306, 306), ('RRset', 306, 306), ('IPv4Address', 303, 257), ('function', 279770, 239), ('cell', 208935, 217), ('IP', 19, 19), ('frozenset', 18984, 17), ('Adapter', 14, 14), ('IPv4Network', 30, 1), ('_SelectorDatagramTransport', 12, 1), ('SystemNetworkData', 1, 1), ('odict_values', 1, 1), ('PTRResolver', 1, 1)]
2024-04-02 11:26:12.284 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('function', 279790, 20), ('frozenset', 18986, 2)]
2024-04-02 11:26:42.007 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 279908, 118), ('cell', 209049, 114), ('frozenset', 18994, 8)]
2024-04-02 11:27:12.357 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 280046, 138), ('cell', 209154, 105), ('frozenset', 19004, 10)]
2024-04-02 11:27:42.081 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 280152, 106), ('cell', 209238, 84), ('tuple', 286378, 14), ('frozenset', 19012, 8)]
2024-04-02 11:28:12.298 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('tuple', 286434, 56), ('function', 280208, 56), ('cell', 209286, 48), ('frozenset', 19016, 4)]
2024-04-02 11:28:42.065 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('function', 280478, 270), ('cell', 209524, 238), ('tuple', 286668, 234), ('dict', 235735, 35), ('frozenset', 19035, 19)]
2024-04-02 11:29:12.287 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('dict', 235823, 88), ('function', 280488, 10), ('tuple', 286677, 9), ('cell', 209526, 2), ('frozenset', 19036, 1)]
2024-04-02 11:29:42.093 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 280674, 186), ('tuple', 286850, 173), ('cell', 209692, 166), ('dict', 235890, 67), ('frozenset', 19049, 13)]
2024-04-02 11:30:12.740 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('function', 280797, 123), ('tuple', 286969, 119), ('cell', 209800, 108), ('deque', 2931, 22), ('frozenset', 19057, 8), ('Template', 430, 6), ('SignalDict', 102, 2), ('AutomationTrace', 40, 2), ('LimitedSizeDict', 13, 1)]
2024-04-02 11:30:42.103 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('cell', 209849, 49), ('function', 280844, 47), ('tuple', 286996, 27), ('frozenset', 19060, 3)]
2024-04-02 11:31:12.296 CRITICAL (SyncWorker_9) [homeassistant.components.profiler] Memory Growth: [('function', 280963, 119), ('cell', 209939, 90), ('dict', 235972, 82), ('tuple', 287062, 66), ('frozenset', 19069, 9)]
2024-04-02 11:31:42.066 CRITICAL (SyncWorker_20) [homeassistant.components.profiler] Memory Growth: [('function', 281136, 173), ('tuple', 287221, 159), ('cell', 210096, 157), ('dict', 236102, 130), ('frozenset', 19081, 12)]
2024-04-02 11:32:12.373 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 281288, 152), ('tuple', 287351, 130), ('cell', 210222, 126), ('dict', 236134, 32), ('frozenset', 19092, 11)]
2024-04-02 11:32:42.024 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('dict', 236301, 167), ('tuple', 287458, 107), ('function', 281387, 99), ('cell', 210309, 87), ('frozenset', 19099, 7), ('uint8_t', 8341, 3)]
2024-04-02 11:33:12.326 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('tuple', 287542, 84), ('cell', 210376, 67), ('function', 281452, 65), ('frozenset', 19104, 5)]
2024-04-02 11:33:41.992 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 281606, 154), ('tuple', 287654, 112), ('cell', 210488, 112), ('dict', 236354, 53), ('frozenset', 19115, 11)]
2024-04-02 11:34:12.305 CRITICAL (SyncWorker_11) [homeassistant.components.profiler] Memory Growth: [('function', 281764, 158), ('cell', 210630, 142), ('tuple', 287793, 139), ('dict', 236404, 50), ('frozenset', 19126, 11)]
2024-04-02 11:34:42.104 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('dict', 236545, 141), ('function', 281862, 98), ('tuple', 287879, 86), ('cell', 210714, 84), ('frozenset', 19133, 7), ('TraceElement', 499, 2)]
2024-04-02 11:35:13.161 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('tuple', 288024, 145), ('function', 281950, 88), ('cell', 210793, 79), ('CaseInsensitiveDict', 563, 39), ('frozenset', 19139, 6), ('deque', 2932, 1)]
2024-04-02 11:35:42.279 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('function', 282100, 150), ('cell', 210918, 125), ('dict', 236629, 84), ('tuple', 288087, 63), ('frozenset', 19150, 11)]
2024-04-02 11:36:12.178 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('function', 282148, 48), ('tuple', 288116, 29), ('cell', 210946, 28), ('dict', 236646, 17), ('frozenset', 19154, 4), ('TraceElement', 501, 2), ('AutomationTrace', 41, 1)]
2024-04-02 11:36:42.025 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 282246, 98), ('tuple', 288210, 94), ('cell', 211030, 84), ('dict', 236694, 48), ('TraceElement', 513, 12), ('frozenset', 19161, 7)]
2024-04-02 11:37:12.294 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 282400, 154), ('tuple', 288345, 135), ('cell', 211162, 132), ('dict', 236816, 122), ('frozenset', 19172, 11)]
2024-04-02 11:37:42.146 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('tuple', 288449, 104), ('function', 282491, 91), ('cell', 211252, 90), ('dict', 236857, 41), ('frozenset', 19178, 6)]
2024-04-02 11:38:12.441 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('dict', 237068, 211), ('function', 282628, 137), ('cell', 211364, 112), ('tuple', 288554, 105), ('frozenset', 19188, 10)]
2024-04-02 11:38:42.253 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('tuple', 288663, 109), ('function', 282718, 90), ('cell', 211438, 74), ('MemoryBIO', 64, 26), ('HttpRequestParser', 30, 21), ('RequestHandler', 43, 17), ('AccessLogger', 43, 17), ('dict', 237083, 15), ('memoryview', 38, 13), ('managedbuffer', 34, 13), ('_SSLProtocolTransport', 32, 13), ('_SSLSocket', 39, 13), ('SSLProtocol', 32, 13), ('SSLObject', 32, 13), ('builtin_function_or_method', 10072, 6), ('frozenset', 19193, 5), ('PosixPath', 1580, 2)]
2024-04-02 11:39:12.241 CRITICAL (SyncWorker_24) [homeassistant.components.profiler] Memory Growth: [('tuple', 289565, 902), ('cell', 211654, 216), ('dict', 237295, 212), ('function', 282827, 109), ('lock', 239, 71), ('builtin_function_or_method', 10123, 51), ('PosixPath', 1618, 38), ('Thread', 70, 36), ('Empty', 64, 35), ('Condition', 114, 32), ('CaseInsensitiveDict', 586, 23), ('traceback', 8941, 16), ('frozenset', 19198, 5), ('defaultdict', 985, 2), ('CacheKey', 47, 1), ('Pattern', 804, 1), ('CachedIPv4Address', 50, 1)]
2024-04-02 11:39:42.376 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 283066, 239), ('cell', 211869, 215), ('dict', 237434, 139), ('frozenset', 19214, 16), ('builtin_function_or_method', 10132, 9), ('ReadAttributeRecord', 4, 4), ('TypeValue', 4, 4), ('CaseInsensitiveDict', 589, 3), ('Condition', 115, 1), ('int16s', 16, 1), ('Read_Attributes_rsp', 1, 1)]
2024-04-02 11:40:12.906 CRITICAL (SyncWorker_25) [homeassistant.components.profiler] Memory Growth: [('function', 283119, 53), ('cell', 211902, 33), ('deque', 2956, 24), ('frozenset', 19218, 4), ('CaseInsensitiveDict', 593, 4)]
2024-04-02 11:40:42.432 CRITICAL (SyncWorker_54) [homeassistant.components.profiler] Memory Growth: [('function', 283161, 42), ('cell', 211938, 36), ('frozenset', 19221, 3), ('Empty', 65, 1), ('traceback', 8942, 1)]
2024-04-02 11:41:12.253 CRITICAL (SyncWorker_15) [homeassistant.components.profiler] Memory Growth: [('function', 283370, 209), ('cell', 212116, 178), ('frozenset', 19236, 15)]
2024-04-02 11:41:42.068 CRITICAL (SyncWorker_40) [homeassistant.components.profiler] Memory Growth: [('dict', 237453, 19)]
2024-04-02 11:42:12.384 CRITICAL (SyncWorker_32) [homeassistant.components.profiler] Memory Growth: [('dict', 237696, 243), ('function', 283607, 237), ('cell', 212337, 221), ('frozenset', 19252, 16), ('builtin_function_or_method', 10136, 4), ('traceback', 8943, 1)]
2024-04-02 11:42:42.257 CRITICAL (SyncWorker_57) [homeassistant.components.profiler] Memory Growth: [('tuple', 289595, 30), ('function', 283634, 27), ('cell', 212358, 21), ('frozenset', 19255, 3)]
2024-04-02 11:43:12.325 CRITICAL (SyncWorker_21) [homeassistant.components.profiler] Memory Growth: [('function', 283635, 1), ('LVBytes', 6, 1)]
2024-04-02 11:43:42.142 CRITICAL (SyncWorker_47) [homeassistant.components.profiler] Memory Growth: [('function', 283900, 265), ('tuple', 289804, 209), ('cell', 212566, 208), ('dict', 237793, 97), ('frozenset', 19274, 19)]
2024-04-02 11:44:12.361 CRITICAL (SyncWorker_8) [homeassistant.components.profiler] Memory Growth: [('dict', 237910, 117), ('tuple', 289871, 67), ('function', 283960, 60), ('cell', 212622, 56), ('frozenset', 19278, 4), ('CaseInsensitiveDict', 595, 2)]
2024-04-02 11:44:42.395 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 284073, 113), ('cell', 212723, 101), ('tuple', 289971, 100), ('dict', 237943, 33), ('frozenset', 19286, 8)]
2024-04-02 11:45:13.075 CRITICAL (SyncWorker_24) [homeassistant.components.profiler] Memory Growth: [('tuple', 290118, 147), ('function', 284209, 136), ('cell', 212833, 110), ('CaseInsensitiveDict', 626, 31), ('frozenset', 19296, 10)]
2024-04-02 11:45:42.156 CRITICAL (SyncWorker_32) [homeassistant.components.profiler] Memory Growth: [('function', 284251, 42), ('cell', 212869, 36), ('frozenset', 19299, 3)]
2024-04-02 11:46:12.482 CRITICAL (SyncWorker_13) [homeassistant.components.profiler] Memory Growth: [('function', 284507, 256), ('cell', 213095, 226), ('tuple', 290328, 210), ('dict', 238105, 162), ('frozenset', 19317, 18)]
2024-04-02 11:46:42.131 CRITICAL (SyncWorker_50) [homeassistant.components.profiler] Memory Growth: [('function', 284577, 70), ('tuple', 290394, 66), ('cell', 213155, 60), ('dict', 238146, 41), ('frozenset', 19322, 5)]
2024-04-02 11:47:12.451 CRITICAL (SyncWorker_28) [homeassistant.components.profiler] Memory Growth: [('dict', 238238, 92), ('tuple', 290438, 44), ('function', 284619, 42), ('cell', 213191, 36), ('frozenset', 19325, 3)]
2024-04-02 11:47:42.210 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('function', 284811, 192), ('tuple', 290605, 167), ('cell', 213349, 158), ('dict', 238268, 30), ('frozenset', 19339, 14)]
2024-04-02 11:48:12.385 CRITICAL (SyncWorker_10) [homeassistant.components.profiler] Memory Growth: [('dict', 238390, 122), ('cell', 213412, 63), ('tuple', 290664, 59), ('function', 284869, 58), ('frozenset', 19343, 4)]
2024-04-02 11:48:42.488 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('function', 285011, 142), ('tuple', 290796, 132), ('cell', 213527, 115), ('dict', 238493, 103), ('frozenset', 19353, 10)]
2024-04-02 11:49:12.388 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 285158, 147), ('tuple', 290939, 143), ('cell', 213661, 134), ('dict', 238618, 125), ('frozenset', 19366, 13), ('builtin_function_or_method', 10140, 4), ('Condition', 119, 4), ('CaseInsensitiveDict', 630, 4), ('RLock', 72, 1), ('OSError', 2, 1), ('lock', 240, 1), ('Template', 431, 1)]
2024-04-02 11:49:42.439 CRITICAL (SyncWorker_26) [homeassistant.components.profiler] Memory Growth: [('function', 285179, 21), ('tuple', 290958, 19), ('cell', 213669, 8)]
2024-04-02 11:50:13.076 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('function', 285407, 228), ('cell', 213871, 202), ('tuple', 291144, 186), ('frozenset', 19381, 15), ('deque', 2969, 13)]
2024-04-02 11:50:42.168 CRITICAL (SyncWorker_27) [homeassistant.components.profiler] Memory Growth: [('function', 285470, 63), ('cell', 213917, 46), ('tuple', 291175, 31), ('frozenset', 19386, 5)]
2024-04-02 11:51:12.444 CRITICAL (SyncWorker_52) [homeassistant.components.profiler] Memory Growth: [('function', 285572, 102), ('cell', 214007, 90), ('tuple', 291242, 67), ('frozenset', 19393, 7)]
2024-04-02 11:51:42.453 CRITICAL (SyncWorker_41) [homeassistant.components.profiler] Memory Growth: [('function', 285722, 150), ('tuple', 291370, 128), ('cell', 214129, 122), ('dict', 238630, 12), ('frozenset', 19404, 11)]
2024-04-02 11:52:12.337 CRITICAL (SyncWorker_35) [homeassistant.components.profiler] Memory Growth: [('dict', 238704, 74), ('tuple', 291440, 70), ('cell', 214197, 68), ('function', 285783, 61), ('frozenset', 19408, 4)]
2024-04-02 11:52:42.653 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('dict', 238872, 168), ('function', 285922, 139), ('tuple', 291565, 125), ('cell', 214306, 109), ('frozenset', 19418, 10)]
2024-04-02 11:53:12.421 CRITICAL (SyncWorker_22) [homeassistant.components.profiler] Memory Growth: [('tuple', 291707, 142), ('cell', 214435, 129), ('function', 286048, 126), ('dict', 238907, 35), ('frozenset', 19427, 9)]
2024-04-02 11:53:42.334 CRITICAL (SyncWorker_28) [homeassistant.components.profiler] Memory Growth: [('dict', 239011, 104), ('function', 286090, 42), ('cell', 214451, 16), ('tuple', 291714, 7), ('frozenset', 19430, 3), ('ZeroconfIPv6Address', 26, 1)]
2024-04-02 11:54:12.523 CRITICAL (SyncWorker_53) [homeassistant.components.profiler] Memory Growth: [('function', 286286, 196), ('tuple', 291885, 171), ('cell', 214619, 168), ('frozenset', 19444, 14), ('dict', 239019, 8)]
2024-04-02 11:54:42.360 CRITICAL (SyncWorker_18) [homeassistant.components.profiler] Memory Growth: [('dict', 239171, 152), ('tuple', 291941, 56), ('function', 286341, 55), ('cell', 214662, 43), ('frozenset', 19448, 4)]
2024-04-02 11:55:13.290 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('tuple', 292015, 74), ('function', 286394, 53), ('cell', 214705, 43), ('CaseInsensitiveDict', 659, 29), ('frozenset', 19452, 4)]
2024-04-02 11:55:42.331 CRITICAL (SyncWorker_40) [homeassistant.components.profiler] Memory Growth: [('function', 286638, 244), ('cell', 214921, 216), ('tuple', 292204, 189), ('frozenset', 19469, 17), ('dict', 239176, 5)]
2024-04-02 11:56:12.624 CRITICAL (SyncWorker_36) [homeassistant.components.profiler] Memory Growth: [('function', 286648, 10), ('cell', 214923, 2), ('frozenset', 19470, 1)]
2024-04-02 11:56:42.473 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('function', 286820, 172), ('tuple', 292360, 156), ('cell', 215077, 154), ('dict', 239297, 121), ('frozenset', 19482, 12)]
2024-04-02 11:57:12.307 CRITICAL (SyncWorker_49) [homeassistant.components.profiler] Memory Growth: [('function', 286918, 98), ('tuple', 292448, 88), ('cell', 215158, 81), ('dict', 239319, 22), ('frozenset', 19489, 7), ('CaseInsensitiveDict', 661, 2)]
2024-04-02 11:57:42.307 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('dict', 239405, 86), ('function', 286972, 54), ('cell', 215207, 49), ('tuple', 292493, 45), ('frozenset', 19493, 4)]
2024-04-02 11:58:12.576 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('function', 287168, 196), ('tuple', 292669, 176), ('cell', 215375, 168), ('dict', 239465, 60), ('frozenset', 19507, 14)]
2024-04-02 11:58:42.267 CRITICAL (SyncWorker_23) [homeassistant.components.profiler] Memory Growth: [('dict', 239564, 99), ('tuple', 292722, 53), ('function', 287220, 52), ('cell', 215413, 38), ('frozenset', 19511, 4)]
2024-04-02 11:59:13.002 CRITICAL (SyncWorker_38) [homeassistant.components.profiler] Memory Growth: [('function', 287324, 104), ('cell', 215514, 101), ('tuple', 292813, 91), ('dict', 239584, 20), ('frozenset', 19518, 7), ('_TrackPointUTCTime', 157, 1)]
2024-04-02 11:59:42.304 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 287488, 164), ('tuple', 292957, 144), ('cell', 215645, 131), ('dict', 239637, 53), ('frozenset', 19530, 12)]
2024-04-02 12:00:13.345 CRITICAL (SyncWorker_56) [homeassistant.components.profiler] Memory Growth: [('cell', 215745, 100), ('function', 287582, 94), ('tuple', 293045, 88), ('frozenset', 19536, 6), ('_TrackPointUTCTime', 160, 3)]
2024-04-02 12:00:42.448 CRITICAL (SyncWorker_17) [homeassistant.components.profiler] Memory Growth: [('function', 287738, 156), ('cell', 215885, 140), ('tuple', 293146, 101), ('dict', 239677, 40), ('frozenset', 19547, 11)]
2024-04-02 12:01:12.485 CRITICAL (SyncWorker_57) [homeassistant.components.profiler] Memory Growth: [('function', 287789, 51), ('cell', 215920, 35), ('tuple', 293178, 32), ('frozenset', 19551, 4)]
2024-04-02 12:01:42.256 CRITICAL (SyncWorker_35) [homeassistant.components.profiler] Memory Growth: [('tuple', 293276, 98), ('function', 287886, 97), ('cell', 215992, 72), ('dict', 239744, 67), ('frozenset', 19558, 7)]
2024-04-02 12:02:12.383 CRITICAL (SyncWorker_35) [homeassistant.components.profiler] Memory Growth: [('dict', 240323, 579), ('tuple', 293514, 238), ('function', 288086, 200), ('cell', 216173, 181), ('partial', 6653, 22), ('HassJob', 4678, 21), ('frozenset', 19572, 14), ('TimerHandle', 2622, 3), ('RegistryEntry', 1803, 1), ('MobileAppSensor', 89, 1)]
2024-04-02 12:02:42.625 CRITICAL (SyncWorker_36) [homeassistant.components.profiler] Memory Growth: [('cell', 216242, 69), ('function', 288144, 58), ('tuple', 293545, 31), ('frozenset', 19576, 4)]
2024-04-02 12:03:12.530 CRITICAL (SyncWorker_32) [homeassistant.components.profiler] Memory Growth: [('tuple', 294584, 1039), ('function', 289107, 963), ('cell', 217098, 856), ('dict', 241164, 841), ('frozenset', 19774, 198), ('OrderedSet', 349, 46), ('BindParameter', 384, 39), ('BinaryExpression', 221, 31), ('Select', 138, 22), ('PyWrapper', 91, 22), ('_anonymous_label', 137, 19), ('_ColumnMetrics', 861, 18), ('Column', 703, 14), ('immutabledict', 389, 13), ('DDLEventsDispatch', 892, 12), ('AnnotatedColumn', 190, 12), ('_truncated_label', 148, 12), ('ResultColumnsEntry', 168, 11), ('_ColumnsPlusNames', 146, 11), ('_ORMColumnEntity', 54, 8), ('default_compile_options', 51, 7), ('ColumnSet', 33, 6), ('ReadOnlyColumnCollection', 86, 6), ('_ListenerCollection', 112, 6), ('ColumnCollection', 44, 6), ('ForeignKey', 40, 6), ('prefix_anon_map', 76, 6), ('AnalyzedFunction', 32, 6), ('LambdaOptions', 32, 6), ('Subquery', 32, 6), ('Comparator', 259, 4), ('BooleanClauseList', 46, 4), ('ColumnClause', 41, 4), ('partial', 6656, 3), ('State', 4892, 3), ('MySQLCompiler_mysqldb', 49, 3), ('CursorResultMetaData', 42, 3), ('ORMSelectCompileState', 31, 3), ('FromLinter', 33, 3), ('StatementLambdaElement', 23, 3), ('_RawColumnEntity', 21, 3), ('LinkedLambdaElement', 9, 3), ('defaultdict', 987, 2), ('Label', 52, 2), ('CacheKey', 49, 2), ('_OffsetLimitParam', 9, 2), ('UnaryExpression', 8, 2), ('CompoundSelect', 8, 2), ('AnnotatedColumnClause', 18, 2)]
2024-04-02 12:03:42.804 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('function', 290477, 1370), ('cell', 218269, 1171), ('tuple', 295745, 1161), ('dict', 242111, 947), ('frozenset', 19872, 98), ('list', 96603, 89), ('State', 4900, 8), ('PosixPath', 1619, 1)]
2024-04-02 12:04:12.878 CRITICAL (SyncWorker_12) [homeassistant.components.profiler] Memory Growth: [('function', 293378, 2901), ('tuple', 298317, 2572), ('cell', 220827, 2558), ('dict', 243689, 1578), ('list', 97074, 471), ('frozenset', 20076, 204), ('builtin_function_or_method', 10189, 49), ('Event', 2194, 19), ('State', 4911, 11), ('AutomationTrace', 42, 1)]
2024-04-02 12:04:43.613 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('function', 297412, 4034), ('dict', 247639, 3950), ('cell', 224228, 3401), ('tuple', 301659, 3342), ('list', 97653, 579), ('frozenset', 20367, 291), ('ReferenceType', 30494, 273), ('State', 4914, 3)]
2024-04-02 12:05:13.589 CRITICAL (SyncWorker_16) [homeassistant.components.profiler] Memory Growth: [('function', 300041, 2629), ('tuple', 303901, 2242), ('cell', 226465, 2237), ('list', 97948, 295), ('frozenset', 20555, 188), ('State', 4915, 1), ('TemplateState', 1819, 1)]
2024-04-02 12:05:43.062 CRITICAL (SyncWorker_49) [homeassistant.components.profiler] Memory Growth: [('function', 304857, 4816), ('tuple', 308057, 4156), ('cell', 230598, 4133), ('dict', 249981, 2342), ('list', 98627, 679), ('frozenset', 20899, 344), ('Context', 7954, 39), ('TimerHandle', 2658, 36)]
2024-04-02 12:06:13.309 CRITICAL (SyncWorker_30) [homeassistant.components.profiler] Memory Growth: [('tuple', 318328, 10271), ('function', 309820, 4963), ('cell', 234972, 4374), ('dict', 253626, 3645), ('list', 100856, 2229), ('frozenset', 21249, 350), ('set', 20716, 102), ('Context', 8010, 56), ('method', 13831, 56), ('builtin_function_or_method', 10227, 38), ('Event', 2226, 32), ('coroutine', 805, 26), ('hamt_bitmap_node', 582, 17), ('generator', 78, 16), ('Task', 145, 16), ('_GeneratorContextManager', 67, 11), ('FutureIter', 87, 7), ('Future', 581, 6), ('hamt', 524, 6), ('defaultdict', 991, 4), ('StopReason', 18, 4), ('list_iterator', 23, 3), ('MultiDictProxy', 26, 2), ('PayloadAccessError', 12, 2), ('TimerNoop', 12, 2), ('AutomationTrace', 44, 2), ('_QueuedScriptRun', 6, 2), ('enumerate', 18, 2), ('URL', 39, 1), ('SplitResult', 165, 1), ('RawRequestMessage', 27, 1), ('immutabledict', 390, 1), ('SimpleCookie', 14, 1), ('CacheKey', 50, 1), ('MultiDict', 16, 1), ('HomeAssistantRequest', 26, 1), ('StreamWriter', 26, 1), ('CIMultiDictProxy', 155, 1), ('UrlMappingMatchInfo', 26, 1), ('_ScriptRun', 14, 1), ('ServiceCall', 6, 1), ('_AsyncGeneratorContextManager', 12, 1), ('async_generator', 12, 1)]
2024-04-02 12:06:43.439 CRITICAL (SyncWorker_32) [homeassistant.components.profiler] Memory Growth: [('function', 313932, 4112), ('cell', 238406, 3434), ('dict', 254719, 1093), ('frozenset', 21546, 297)]
2024-04-02 12:07:13.338 CRITICAL (SyncWorker_52) [homeassistant.components.profiler] Memory Growth: [('function', 315470, 1538), ('cell', 239717, 1311), ('dict', 255070, 351), ('frozenset', 21658, 112), ('State', 4916, 1), ('BindParameter', 385, 1), ('StatementLambdaElement', 24, 1), ('LambdaOptions', 33, 1), ('callable_iterator', 1, 1)]
2024-04-02 12:07:43.411 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 317407, 1937), ('cell', 241381, 1664), ('dict', 255921, 851), ('tuple', 318886, 558), ('frozenset', 21794, 136), ('State', 4924, 8)] ```

alexh3o commented 5 months ago

Edit: Sorry, below message was for profiler.start_log_objects_sources I had provided profiler.start_log_objects with 4 hours of log in my initial post. I'll re-run it with 30s interval and post again

As soon as I start profiler.start_log_objects_sources it crashes HA Core. I just get the start in the core logs:

2024-04-02 13:33:43.710 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] New object list (0/84207) at ["{'__name__': 'gc', '__doc__': 'This module provides access to the garbage collector for reference cycles.\\n\\nenable() -- Enable automatic garbage collection.\\ndisable() -- Disable automatic garbage collection.\\nisenabled() -- Returns true if automatic collection is enabled.\\ncollect() -- Do a full collection right now.\\nget_count() -- Return the current collection counts.\\nget_stats() -- Return list of dictionaries containing per-generation stats.\\nset_debug() -- Set debugging flags.\\nget_debug() -- Get debugging flags.\\nset_threshold() -- Set the collection thresholds.\\nget_threshold() -- Return the current the collection thresholds.\\nget_objects() -- Return a list of all objects tracked by the collector.\\nis_tracked() -- Returns true if a given object is tracked.\\nis_finalized() -- Returns true if a given object has been already finalized.\\nget_referrers() -- Return the list of objects that refer to an object.\\nget_referents() -- Return the list of objects that an object refers to.\\nfreeze() -- Freeze all tracked objects and ignore them for future collections.\\nunfreeze() -- Unfreeze all objects in the permanent generation.\\nget_freeze_count() -- Return the number of objects in the permanent generation.\\n', '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='gc', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in'), 'enable': <built-in function enable>, 'disable': <built-in function disable>, 'isenabled': <built-in function isenabled>, 'set_debug': <built-in function set_debug>, 'get_debug': <built-in function get_debug>, 'get_count': <built-in function get_count>, 'set_threshold': <built-in function set_threshold>, 'get_threshold': <built-in function get_threshold>, 'collect': <built-in function collect>, 'get_objects': <built-in function get_objects>, 'get_stats': <built-in function get_stats>, 'is_tracked': <built-in function is_tracked>, 'is_finalized': <built-in function is_finalized>, 'get_referrers': <built-in function get_referrers>, 'get_referents': <built-in function get_referents>, 'freeze': <built-in function freeze>, 'unfreeze': <built-in function unfreeze>, 'get_freeze_count': <built-in function get_freeze_count>, 'garbage': [], 'callbacks': [], 'DEBUG_STATS': 1, 'DEBUG_COLLECTABLE': 2, 'DEBUG_UNCOLLECTABLE': 4, 'DEBUG_SAVEALL': 32, 'DEBUG_LEAK': 38}", '[]']: []
2024-04-02 13:33:44.093 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] New object list (0/84207) at ["{'__name__': 'gc', '__doc__': 'This module provides access to the garbage collector for reference cycles.\\n\\nenable() -- Enable automatic garbage collection.\\ndisable() -- Disable automatic garbage collection.\\nisenabled() -- Returns true if automatic collection is enabled.\\ncollect() -- Do a full collection right now.\\nget_count() -- Return the current collection counts.\\nget_stats() -- Return list of dictionaries containing per-generation stats.\\nset_debug() -- Set debugging flags.\\nget_debug() -- Get debugging flags.\\nset_threshold() -- Set the collection thresholds.\\nget_threshold() -- Return the current the collection thresholds.\\nget_objects() -- Return a list of all objects tracked by the collector.\\nis_tracked() -- Returns true if a given object is tracked.\\nis_finalized() -- Returns true if a given object has been already finalized.\\nget_referrers() -- Return the list of objects that refer to an object.\\nget_referents() -- Return the list of objects that an object refers to.\\nfreeze() -- Freeze all tracked objects and ignore them for future collections.\\nunfreeze() -- Unfreeze all objects in the permanent generation.\\nget_freeze_count() -- Return the number of objects in the permanent generation.\\n', '__package__': '', '__loader__': <class '_frozen_importlib.BuiltinImporter'>, '__spec__': ModuleSpec(name='gc', loader=<class '_frozen_importlib.BuiltinImporter'>, origin='built-in'), 'enable': <built-in function enable>, 'disable': <built-in function disable>, 'isenabled': <built-in function isenabled>, 'set_debug': <built-in function set_debug>, 'get_debug': <built-in function get_debug>, 'get_count': <built-in function get_count>, 'set_threshold': <built-in function set_threshold>, 'get_threshold': <built-in function get_threshold>, 'collect': <built-in function collect>, 'get_objects': <built-in function get_objects>, 'get_stats': <built-in function get_stats>, 'is_tracked': <built-in function is_tracked>, 'is_finalized': <built-in function is_finalized>, 'get_referrers': <built-in function get_referrers>, 'get_referents': <built-in function get_referents>, 'freeze': <built-in function freeze>, 'unfreeze': <built-in function unfreeze>, 'get_freeze_count': <built-in function get_freeze_count>, 'garbage': [], 'callbacks': [], 'DEBUG_STATS': 1, 'DEBUG_COLLECTABLE': 2, 'DEBUG_UNCOLLECTABLE': 4, 'DEBUG_SAVEALL': 32, 'DEBUG_LEAK': 38}", '[]']: []

Supervisor gives the following error:

2024-04-02` 13:34:25.970 WARNING (MainThread) [supervisor.homeassistant.core] Watchdog found Home Assistant failed, restarting...`

This is with HA OS in proxmox on an N100 Mini PC, and I have the same crash after starting the profiler logging in normal and safe mode. On the raspberry pi that I had a few days ago with only HAOS, i didn't try that profiler commande, but the profiler:memory command had HA Core crash as well.

alexh3o commented 5 months ago

Where can I find the log file after running profiler.start_log_objects ?

In Settings / System / Logs / Home Assistant Logs, and click "Download full log" at the bottom

alexh3o commented 5 months ago

Here is my log. 1.5 hours with 30 seconds interval, this is with HAOS in proxmox on a mini-PC, in normal mode. In my initial post there is also a 4 hours profiler log with HAOS alone on a rpi4, in safe mode.

202404021540_HACoreMemoryLeak-N100ProxmoxHAOS-NormalMode.log

This is the matching memory increase

Edit: Looking at my log file, i see NodeListClass, NodeStrClass and NodeDictClass seem to be the issue? I could provide a dump log objects of these 3, but there is a lot of private information in it, i've edited them my best but that i would rather send it by pm if necessary. Those 3 seem to contain all my entities and yaml from frontend, configuration, automations, etc... and repeated over time with exact same lines

woniol commented 5 months ago

Here is my log: home-assistant_2024-04-02T13-04-28.155Z.log

bdraco commented 5 months ago

@tygozwolle

Here is my log

2024-04-02 12:07:43.411 CRITICAL (SyncWorker_19) [homeassistant.components.profiler] Memory Growth: [('function', 317407, 1937), ('cell', 241381, 1664), ('dict', 255921, 851), ('tuple', 318886, 558), ('frozenset', 21794, 136), ('State', 4924, 8)] ```

Your leak is something creating frozenset. You might be able to dump them https://www.home-assistant.io/integrations/profiler/#service-profilerdump_log_objects to figure out whats in them. Do it soon after restart before the leak it too big

Your leak is different than than alexh3o's leak so you should start a new issue as well because once alexh3o's issue is solved this issue will close

bdraco commented 5 months ago

@alexh3o

Here is my log. 1.5 hours with 30 seconds interval, this is with HAOS in proxmox on a mini-PC, in normal mode. In my initial post there is also a 4 hours profiler log with HAOS alone on a rpi4, in safe mode.

202404021540_HACoreMemoryLeak-N100ProxmoxHAOS-NormalMode.log

It looks like your system is leaking ReceiveMessage You might be able to dump them https://www.home-assistant.io/integrations/profiler/#service-profilerdump_log_objects to figure out whats in them. Do it soon after restart before the leak it too big

bdraco commented 5 months ago

@woniol

Here is my log: home-assistant_2024-04-02T13-04-28.155Z.log

There are no obvious leaks in your log so you have a different issue as well

home-assistant[bot] commented 5 months ago

profiler documentation profiler source

home-assistant[bot] commented 5 months ago

Hey there @emontnemery, @jbouwh, mind taking a look at this issue as it has been labeled with an integration (mqtt) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `mqtt` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign mqtt` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

mqtt documentation mqtt source _{^{(message by IssueLinks)}}

Tygozwolle commented 5 months ago

What program do you use to view te logs, to check where the leak is?

alexh3o commented 5 months ago

Hello @bdraco , thank you for your answer.

Could you have a second look ?

If I understand the profiler logs correctly, there was a small memory growth on ReceiveMessagein the log I provided, showing last, but not significant compared to the memory growth I have on NodeStrClass, NodeListClassand NodeDictClass:

Those show every 15 minutes only, but repetedly and always with the same huge growth (delta). I've attached new logs, the three I've mentioned are still showing, but not ReceiveMessage

I have made a dump of the three Node___Class, but I find all my entities, automations, frontend yaml in them so I can't make sense of it.

I have 200 000 lines with stuff like this:

I need to review my triggers that happen every /15 minutes, although there was no problem prior to march update, and I believe many stuff in the dump is not related to a 15 minutes trigger.

202404031358 Profiler Log alexh3o.txt

bdraco commented 5 months ago

but not significant compared to the memory growth I have on NodeStrClass, NodeListClassand NodeDictClass:

I've already fixed that one in 2024.4.x via https://github.com/home-assistant/core/pull/114602. I haven't fixed the ReceiveMessage one

alexh3o commented 5 months ago

Thanks, updating to 2024.4 did remove 94% of the memory leak (now around 6MB/hour against 100MB before) I saw the leak on ReceiveMessage will be fixed next month, which is good. Appart from this, there might be a small leak remaining on deque too, i can post logs if necessary.

bdraco commented 5 months ago

Great. Would you please open a new issue for the deque with the dump. Since deques are used in a lot of place it will take more time to find. If you could do the dump at startup, 1h, and 2h we might be able to figure out what is in them so we can figure out where they are being created

alexh3o commented 5 months ago

So I've let HA run longer and ran a profiler log for 9 hours, in normal mode (not safe mode) : there was actually no visible memory use increase on HA Core.

In the profiler logs for the most visible types I have the following occurences and growth for those 9 hours (60 seconds profiler interval, 534 log lines) :

type : occurences, start qty => end Qty (qty increase)
ReceiveMessage: 43, 5355 => 7480 (+40%)
TraceElement: 80, 4979 => 6044 (+21%)
deque: 66, 8473 => 9601 (+13%)
dict: 131, 152104 =>159001 ( +4%)
list: 63, 98719 =>102055 ( +3%)

So I'll watch it over time, but considering the impact on actual memory usage seems null, and ReceiveMessage will be addressed, my current feeling/understanding is all's good and there's no new issue to open.

bdraco commented 5 months ago

It looks like the deque leak is so small that it might be only be a problem if you are running for 100s days, or it might level out after a week and turn out to not be an issue.

That one is going to be very hard to track down if it is a problem as I couldn't replicate it on any of my systems

alexh3o commented 5 months ago

Yes I totally agree, the intent of my message was mostly to explain why I wasn't opening a new issue. Thanks again for the fixes 👍

home-assistant / core