frenck
commented
3 years ago There is not much information in this issue to work with, which makes it kinda hard to come up with a cause or solution.
Is there anything visible on the VM console?
Closed danygi79 closed 2 years ago
frenck
commented
3 years ago There is not much information in this issue to work with, which makes it kinda hard to come up with a cause or solution.
Is there anything visible on the VM console?
throwawat
commented
3 years ago Oh I think I have this issue but I think I found the cause generally. I allocated 12gb of ram just to see what would happen and the result is a VERY steady ever increasing ram usage (While I'm not really doing anything). it starts at 2gigs of usage but from there it slowly creeps up. over half an hour its gone up to 9.5gbs
I think we have a memory leak.
I'm running under proxmox with core-2021.10.3 / supervisor-2021.10.0 and the core reports using over 57% of the 12gb of ram and increasing. it keeps filling up the ram until I start getting out of memory errors and everything stops working until I reset the VM
Edit:
Hmmm I think I might have found the problem one of my automations uses the 'Until' condition turning that off seems to stop the leak. While the until is active is also increases my CPU usage from like 1-2% with occasional bursts to around a constant 30%
This same automation was not doing this on core-2021.9.x
thecode
commented
3 years ago Edit:
Hmmm I think I might have found the problem one of my automations uses the 'Until' condition turning that off seems to stop the leak. While the until is active is also increases my CPU usage from like 1-2% with occasional bursts to around a constant 30%
This same automation was not doing this on core-2021.9.x
@throwawat, I am not sure you have the same issue as @danygi79, would be better to open a new issue "Automation causes memory leak ...", fill in all the details and attach logs.
dennidoubles
commented
3 years ago Experiencing a memory leak also, after a long period of stable memory usage. Running HASSIO in a VM with 6GB on Unraid. Core version 10.1, Supervisor 10.0 Allocated 8GB now, but that won't overcome any issue I guess.
A snap from my memory usage:
A snap from my Unraid/VM log:
A snap from my CLI:
Don't see any strange things happening in my HA Logfile, except some connectivity issues with Reolink.
Any advice?
jryan82
commented
3 years ago I'm having a similar issue. Mine is crashing due to running out of memory every night. It runs fine for about 12 hours then crashes. I can't find anything in the logs but I may not be using the right log viewer screen.
I've tried giving it more memory and disabling all my add-ins from HACS but still no luck.
Happening in Version 10.1 and 10.2.
dennidoubles
commented
3 years ago So I've added more GB, but -of course- this is not problem solving. It is climbing up again:
Also processor usage is climbing up again, this is -somehow- related to each other I assume:
HA is still running, but I guess it will stop running within a day or 2 again, if I look at the graphs.
Don't know what to do, instead of restarting every night?
jryan82
commented
3 years ago Updated core to 2021.10.4 today as well as Home Assistant 6.5. Came home and found HA unresponsive again. Out of memory
jryan82
commented
3 years ago Ok, didn't even last 5 hours this time. System failed already, out of memory. :(
throwawat
commented
3 years ago yeah I'd suggest checking your automations to see if turning some of them off help mitigate the issue.
Pirol62
commented
3 years ago At least I would check if an automation may never come back from its wait status. Another typical issue I had in the past is, that an automation had been started randomly too often. I checked and redefined all my automation triggers and sensors in order to have control about the execution frequency.
gregoinc
commented
3 years ago I was experiencing this issue some days back, but thought it had gone... and overnight this happened again.
I know there has been a few updates since the issue was recognised. Is there likely to be more, or should I consider restoring back to a working config?
Latest memory stats... but eerily stable now?
Did some more experimenting, and found when I added a temperature sensor to the lovelace overview page, this is hwta happened to the memory usage...
Here's the sensor I added.
So there appears to be a memory leak associated with adding the sensor?
cad1231
commented
3 years ago Wanted to note that I have the same issue. I run hass supervised on a proxmox vm, and it crashes every few hours after upgrading to 2021.10, including 2021.10.4. I would need to either reboot the whole node or log into ssh and issue the 'ha core restart' command to get it working again. If I went to the supervisor port (4357), it would just say that the supervisor is connected. But port 8123 would fail to connect and I would not be able to access the web UI. I do not have any sensor cards added to the lovelace UI, but I do have temperature sensors as part of quick glace and entities cards.
I downgraded back to 2021.9.7, and it has been stable for over 24 hours now. There is clearly some kind of issue with 2021.10. I am happy to upgrade to it for the purposes of pulling logs, but please tell me which logs you would need. Thanks!
gregoinc
commented
3 years ago Wanted to note that I have the same issue. I run hass supervised on a proxmox vm, and it crashes every few hours after upgrading to 2021.10, including 2021.10.4. I would need to either reboot the whole node or log into ssh and issue the 'ha core restart' command to get it working again. If I went to the supervisor port (4357), it would just say that the supervisor is connected. But port 8123 would fail to connect and I would not be able to access the web UI. I do not have any sensor cards added to the lovelace UI, but I do have temperature sensors as part of quick glace and entities cards.
I downgraded back to 2021.9.7, and it has been stable for over 24 hours now. There is clearly some kind of issue with 2021.10. I am happy to upgrade to it for the purposes of pulling logs, but please tell me which logs you would need. Thanks!
Interested to know how you rolled back to 2021.9.7 as I am thinking that is the only way to fix this issue having not seen any other solutions coming from the HA team.
cad1231
commented
3 years ago Wanted to note that I have the same issue. I run hass supervised on a proxmox vm, and it crashes every few hours after upgrading to 2021.10, including 2021.10.4. I would need to either reboot the whole node or log into ssh and issue the 'ha core restart' command to get it working again. If I went to the supervisor port (4357), it would just say that the supervisor is connected. But port 8123 would fail to connect and I would not be able to access the web UI. I do not have any sensor cards added to the lovelace UI, but I do have temperature sensors as part of quick glace and entities cards. I downgraded back to 2021.9.7, and it has been stable for over 24 hours now. There is clearly some kind of issue with 2021.10. I am happy to upgrade to it for the purposes of pulling logs, but please tell me which logs you would need. Thanks!
Interested to know how you rolled back to 2021.9.7 as I am thinking that is the only way to fix this issue having not seen any other solutions coming from the HA team.
Log into ssh and issue 'ha core update --version 2021.9.7'. Alternatively, if you take snapshots, you can just restore from an old snapshot.
bdraco
commented
3 years ago Please use the profiler to get a callgrind.out.XXX file and post it here https://www.home-assistant.io/integrations/profiler/
Additionally, make a py-spy recording and post the SVG file https://github.com/benfred/py-spy
cad1231
commented
3 years ago @bdraco I confess I have never used these tools before - not a Python dev. Can you please give me some more details? The profiler integration appears to run for 60 seconds at a time by default. Is that enough? Should I set it to something else? How do I install py-spy in a supervised environment? Do in install it into the core container?
bdraco
commented
3 years ago The profiler integration appears to run for 60 seconds at a time by default. Is that enough?
That's probably more than long enough if the memory leak has a cpu time impact.
How do I install py-spy in a supervised environment? Do in install it into the core container?
cad1231
commented
3 years ago The profiler integration appears to run for 60 seconds at a time by default. Is that enough?
That's probably more than long enough if the memory leak has a cpu time impact.
How do I install py-spy in a supervised environment? Do in install it into the core container?
Thanks! I'll post either later tonight or tomorrow.
cad1231
commented
3 years ago @bdraco please see attached profile results.zip
I should add, this is on core 2021.10.5. I haven't tried this version before (last one I tried was .4), and it may be that whatever the issue was got fixed in this release. Or you will look at the results and see an obvious problem :)
Thanks for the help!
bdraco
commented
3 years ago Most of your event loop is spent on the brother integration. We should probably add a guard to prevent probing the device if its already been configured
Most of the py-spy time appears to be a camera stream worker.
Can you do a py-spy top for 5 minutes as well?
gregoinc
commented
3 years ago The profiler integration appears to run for 60 seconds at a time by default. Is that enough?
That's probably more than long enough if the memory leak has a cpu time impact.
How do I install py-spy in a supervised environment? Do in install it into the core container?
Thanks! I'll post either later tonight or tomorrow.
How did you find the profiler data? I've added the integration, but not sure where the data is located?
cad1231
commented
3 years ago @bdraco here is the result of py-spy top --pid [core process] after running for about 5 minutes:
Where it says "No such file or directory" flips between that and "failed to suspend process".
I can also confirm that 2021.10.5 did indeed crash overnight, so it did not fix the issue.
@gregoinc the output will be in the /config directory.
bdraco
commented
3 years ago Try disabling the stream integration
If that doesn't do it, please start logging objects and look for which ones are growing
https://www.home-assistant.io/integrations/profiler/#service-profilerstart_log_objects
cad1231
commented
3 years ago @bdraco I can disable it for testing purposes to see if it crashes, but hass without my cameras is really not ideal. I have four rtsp feeds coming into hass that I use on a regular basis.
For testing purposes, is it enough for me to just comment out my camera entries in my configuration.yaml file, or do I need to go further than that?
bdraco
commented
3 years ago Disabling the cameras should be enough
cad1231
commented
3 years ago OK, thanks. I turned off the cameras (commented them out in my configuration.yaml) and added a memory monitoring sensor. I can see memory usage slowly creeping up over the last hour or so. This seems indicative of a continued memory leak somewhere. It'll likely crash in 12-20 hours from now.
I started logging objects, will let it run for an hour or so and will post the result.
gregoinc
commented
3 years ago See profiler results attached... profile.1634423457188610.results.zip
And the Profiler Start log objects... Profiler-Start-log-objects.zip Profiler-Start-log-objects-v1.zip
And visual memory use at time of taking above...
cad1231
commented
3 years ago @bdraco I left the object logging going for a few hours, and this is the result. Not sure it's all that helpful, but you may see something that I don't.
2021-10-16 14:53:45 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('function', 72959, 72959), ('dict', 58652, 58652), ('tuple', 55856, 55856), ('cell', 33523, 33523), ('list', 30433, 30433), ('weakref', 15597, 15597), ('type', 8487, 8487), ('getset_descriptor', 7361, 7361), ('builtin_function_or_method', 5936, 5936), ('set', 5146, 5146), ('property', 4484, 4484), ('member_descriptor', 3451, 3451), ('method', 3229, 3229), ('ModuleSpec', 3214, 3214), ('module', 3208, 3208), ('SourceFileLoader', 3037, 3037), ('wrapper_descriptor', 2880, 2880), ('method_descriptor', 2423, 2423), ('Schema', 2227, 2227), ('frozenset', 1634, 1634), ('PosixPath', 1577, 1577), ('staticmethod', 1314, 1314), ('OrderedDict', 1203, 1203), ('PathMetadata', 1171, 1171), ('DistInfoDistribution', 1170, 1170), ('classmethod', 1085, 1085), ('_abc_data', 1014, 1014), ('LRItem', 1009, 1009), ('ABCMeta', 951, 951), ('Optional', 886, 886), ('RepositoryData', 880, 880), ('RepositoryContent', 880, 880), ('RepositoryInformation', 880, 880), ('RepositoryStatus', 880, 880), ('Validate', 880, 880), ('RepositoryReleases', 880, 880), ('RepositoryVersions', 880, 880), ('RepositoryPath', 880, 880), ('HacsManifest', 880, 880), ('Parameter', 874, 874), ('_tuplegetter', 822, 822), ('HassJob', 622, 622), ('Required', 588, 588), ('HacsIntegrationRepository', 580, 580), ('_GenericAlias', 565, 565), ('mappingproxy', 500, 500), ('FileFinder', 495, 495), ('Logger', 495, 495), ('Context', 454, 454), ('deque', 436, 436), ('_UnionGenericAlias', 410, 410), ('TraversibleType', 401, 401), ('_Listener', 401, 401), ('weakproxy', 401, 401), ('Integration', 396, 396), ('itemgetter', 384, 384), ('Field', 353, 353), ('PointerType', 353, 353), ('MovedAttribute', 346, 346), ('Attribute', 341, 341), ('All', 337, 337), ('FieldDescriptor', 336, 336), ('Production', 334, 334), ('ForwardRef', 333, 333), ('_FieldProperty', 332, 332), ('InstanceState', 324, 324), ('MiniProduction', 310, 310), ('CType', 305, 305), ('RepeatedCompositeFieldContainer', 304, 304), ('State', 292, 292), ('attrgetter', 285, 285), ('FieldDescriptorProto', 238, 238), ('HacsPluginRepository', 236, 236), ('Event', 222, 222), ('TagMap', 218, 218), ('OidOrderedDict', 208, 208), ('And', 205, 205), ('partial', 198, 198), ('ResourceRoute', 195, 195), ('Service', 189, 189), ('_EmptyListener', 188, 188), ('CField', 180, 180), ('MovedModule', 179, 179), ('_CallableGenericAlias', 178, 178), ('NullMessageListener', 178, 178), ('AttributeEvent', 171, 171), ('cython_function_or_method', 170, 170), ('UnicodeValueChecker', 164, 164), ('Events', 163, 163), ('States', 160, 160), ('memoized_property', 157, 157), ('symbol', 156, 156), ('RegistryEntry', 156, 156), ('Packet_metaclass', 153, 153), ('FunctionPtrType', 150, 150), ('RGBColor', 149, 149), ('EnumMeta', 146, 146), ('ByteField', 143, 143), ('Signature', 136, 136), ('status', 131, 131)]
2021-10-16 14:54:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('weakref', 15606, 9), ('TimerHandle', 70, 4), ('memoryview', 5, 3), ('HassJob', 624, 2), ('managedbuffer', 3, 1), ('builtin_function_or_method', 5937, 1), ('PosixPath', 1578, 1), ('LogEntry', 15, 1), ('FileResponse', 3, 1)]
2021-10-16 14:54:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('weakref', 15634, 28), ('tuple', 55866, 10), ('Context', 463, 9), ('cell', 33532, 9), ('TimerHandle', 77, 7), ('list', 30438, 5), ('Handle', 34, 4), ('SelectorKey', 28, 2), ('TaskStepMethWrapper', 1, 1), ('Future', 56, 1), ('frozenset', 1635, 1), ('partial', 199, 1), ('socket', 65, 1), ('suppress', 4, 1), ('ResponseHandler', 2, 1), ('ConnectionKey', 2, 1), ('Quoter', 1, 1), ('ServerDisconnectedError', 1, 1)]
2021-10-16 14:55:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('cell', 33535, 3), ('DNSQuestion', 3, 2), ('State', 293, 1), ('HassJob', 625, 1), ('TimerHandle', 78, 1), ('WSMessage', 5, 1)]
2021-10-16 14:55:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 82, 4)]
2021-10-16 14:56:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('DNSNsec', 13, 1)]
2021-10-16 14:56:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('DNSPointer', 74, 11), ('BufferedReader', 4, 1), ('HTTPResponse', 3, 1), ('SocketIO', 2, 1)]
2021-10-16 14:57:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('DNSService', 33, 3), ('DNSAddress', 12, 2), ('DNSNsec', 14, 1)]
2021-10-16 14:57:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('DNSService', 37, 4)]
2021-10-16 14:58:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('list', 30443, 5), ('HTTPSHandler', 2, 1), ('OpenerDirector', 2, 1), ('UnknownHandler', 2, 1), ('HTTPHandler', 2, 1), ('HTTPDefaultErrorHandler', 2, 1), ('HTTPRedirectHandler', 2, 1), ('FTPHandler', 2, 1), ('FileHandler', 2, 1), ('HTTPErrorProcessor', 2, 1), ('DataHandler', 2, 1)]
2021-10-16 14:58:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 14:59:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('StreamReader', 9, 1), ('SSLProtocol', 2, 1), ('_SSLProtocolTransport', 2, 1), ('_SSLPipe', 2, 1), ('SSLObject', 2, 1), ('HttpResponseParser', 2, 1), ('TimerContext', 3, 1)]
2021-10-16 14:59:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:00:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('weakref', 15657, 23), ('DNSQuestion', 10, 7), ('set', 5147, 1)]
2021-10-16 15:00:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:01:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:01:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:02:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 84, 2)]
2021-10-16 15:02:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 87, 3)]
2021-10-16 15:03:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('function', 72962, 3)]
2021-10-16 15:03:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('tuple', 55870, 4), ('FlagValue', 35, 3), ('Context', 464, 1), ('Ether', 2, 1), ('IP', 2, 1), ('UDP', 2, 1), ('BOOTP', 2, 1), ('DHCP', 2, 1)]
2021-10-16 15:04:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:04:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('DNSPointer', 77, 3)]
2021-10-16 15:05:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('list_iterator', 8, 1)]
2021-10-16 15:05:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:06:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:06:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 90, 3)]
2021-10-16 15:07:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:07:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('HTTPMessage', 2, 1)]
2021-10-16 15:08:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:08:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:09:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:09:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:10:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:10:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('_CipherContext', 1, 1), ('__CDataGCP', 1, 1)]
2021-10-16 15:11:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 96, 6), ('Context', 468, 4)]
2021-10-16 15:11:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:12:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:12:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:13:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('Context', 482, 14), ('TimerHandle', 103, 7), ('MimeType', 10, 1)]
2021-10-16 15:13:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:14:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('traceback', 2, 2), ('timeout', 3, 1)]
2021-10-16 15:14:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:15:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:15:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:16:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:16:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:17:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:17:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 105, 2)]
2021-10-16 15:18:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:18:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('_CipherContext', 2, 1)]
2021-10-16 15:19:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:19:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('mappingproxy', 501, 1), ('State', 294, 1)]
2021-10-16 15:20:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:20:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:21:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:21:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:22:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:22:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:23:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:23:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:24:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:24:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:25:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:25:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:26:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:26:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('TuyaMessage', 2, 2)]
2021-10-16 15:27:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:27:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:28:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('traceback', 5, 3), ('KeyError', 3, 3)]
2021-10-16 15:28:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('LockOperationActivity', 2, 1)]
2021-10-16 15:29:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('DNSText', 43, 1)]
2021-10-16 15:29:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('TimerHandle', 113, 8), ('Context', 484, 2), ('ResponseHandler', 3, 1), ('ConnectionKey', 3, 1)]
2021-10-16 15:30:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:30:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('set', 5153, 6), ('DNSQuestion', 16, 6)]
2021-10-16 15:31:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:31:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:32:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:32:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:33:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('TaskStepMethWrapper', 2, 1), ('coroutine_wrapper', 1, 1), ('ClientTimeout', 3, 1), ('TimeoutHandle', 2, 1), ('ClientRequest', 2, 1), ('CeilTimeout', 1, 1), ('_TransportPlaceholder', 1, 1)]
2021-10-16 15:33:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:34:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:34:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:35:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:35:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:36:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:36:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:37:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:37:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:38:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:38:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:39:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:39:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('dict', 59032, 380), ('AIOGitHubAPIRepositoryTreeContent', 324, 275), ('IPv4Address', 309, 256), ('list', 30651, 208), ('Name', 71, 57), ('RRset', 45, 45), ('QueryMessage', 33, 33), ('tuple', 55892, 22), ('IP', 23, 21), ('Adapter', 18, 18), ('Context', 499, 15), ('PTR', 12, 12), ('deque', 444, 8), ('AIOGitHubAPIRepositoryRelease', 10, 5), ('StreamReader', 13, 4), ('SSLProtocol', 6, 4), ('ResponseHandler', 7, 4), ('_SSLProtocolTransport', 6, 4), ('_SSLPipe', 6, 4), ('SSLObject', 6, 4), ('HttpResponseParser', 6, 4), ('TimerContext', 7, 4), ('socket', 68, 3), ('TransportSocket', 31, 3), ('KeyedRef', 77, 2), ('_SelectorSocketTransport', 22, 2), ('TimerHandle', 114, 1), ('_GenericAlias', 566, 1), ('IPv4Network', 34, 1), ('SelectorKey', 29, 1), ('OrderedDict', 1204, 1), ('list_iterator', 9, 1), ('_SelectorDatagramTransport', 9, 1), ('AIOGitHubAPIClient', 3, 1), ('AIOGitHubAPIRateLimit', 3, 1), ('ConnectionKey', 4, 1), ('AIOGitHubAPIRepository', 2, 1), ('AIOGitHubAPIRepositoryTraffic', 2, 1), ('PTRResolver', 1, 1), ('SystemNetworkData', 1, 1), ('AIOGitHubAPIResponse', 2, 1), ('odict_values', 1, 1), ('_GatheringFuture', 1, 1)]
2021-10-16 15:40:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:40:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:41:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:41:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:42:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:42:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:43:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:43:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:44:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('DNSAddress', 13, 1)]
2021-10-16 15:44:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:45:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:45:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:46:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:46:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:47:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:47:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:48:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:48:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:49:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:49:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:50:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:50:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:51:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:51:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:52:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:52:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:53:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:53:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:54:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('Semaphore', 31, 1)]
2021-10-16 15:54:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:55:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:55:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:56:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:56:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:57:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:57:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:58:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:58:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:59:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 15:59:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:00:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:00:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:01:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:01:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:02:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:02:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:03:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:03:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:04:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:04:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:05:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:05:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:06:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:06:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:07:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:07:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('DNSPointer', 78, 1)]
2021-10-16 16:08:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:08:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:09:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:09:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:10:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:10:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('weakref', 15659, 2)]
2021-10-16 16:11:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:11:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:12:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:12:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:13:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:13:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:14:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:14:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:15:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:15:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:16:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:16:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:17:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('DNSPointer', 88, 10)]
2021-10-16 16:17:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:18:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:18:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('partial', 200, 1)]
2021-10-16 16:19:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:19:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:20:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:20:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:21:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:21:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:22:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:22:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:23:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:23:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:24:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:24:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:25:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:25:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:26:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:26:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:27:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:27:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:28:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:28:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:29:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:29:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:30:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:30:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:31:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('DNSOutgoing', 1, 1), ('list_iterator', 10, 1)]
2021-10-16 16:31:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:32:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:32:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:33:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:33:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:34:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('partial', 201, 1)]
2021-10-16 16:34:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:35:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:35:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:36:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:36:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:37:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:37:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:38:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:38:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('Connection', 2, 1), ('RequestInfo', 2, 1), ('ClientResponse', 2, 1)]
2021-10-16 16:39:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:39:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('tuple', 55926, 34), ('dict', 59057, 25), ('list', 30663, 12), ('Name', 77, 6), ('cell', 33540, 5), ('RRset', 50, 5), ('QueryMessage', 37, 4), ('CastMessage', 1, 1), ('NullMessageListener', 179, 1), ('_Listener', 402, 1), ('weakproxy', 402, 1), ('IPv4Address', 310, 1), ('MimeType', 11, 1), ('PTR', 13, 1)]
2021-10-16 16:40:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:40:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:41:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:41:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:42:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:42:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:43:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:43:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:44:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:44:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:45:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:45:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:46:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:46:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:47:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:47:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:48:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:48:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('StopIteration', 1, 1)]
2021-10-16 16:49:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:49:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:50:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:50:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:51:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:51:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:52:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:52:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:53:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:53:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:54:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:54:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:55:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:55:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:56:16 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:56:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:57:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:57:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:58:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:58:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:59:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 16:59:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:00:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:00:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:01:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:01:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:02:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:02:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:03:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:03:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:04:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:04:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:05:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:05:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:06:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:06:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('builtin_function_or_method', 5938, 1)]
2021-10-16 17:07:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:07:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:08:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:08:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('coroutine', 100, 14), ('frame', 109, 13), ('partial', 203, 2), ('IncompleteReadError', 1, 1)]
2021-10-16 17:09:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:09:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('weakref', 15667, 8)]
2021-10-16 17:10:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:10:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:11:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:11:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('builtin_function_or_method', 5939, 1)]
2021-10-16 17:12:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('TaskStepMethWrapper', 4, 2)]
2021-10-16 17:12:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:13:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:13:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:14:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:14:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('TaskStepMethWrapper', 5, 1)]
2021-10-16 17:15:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:15:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:16:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('weakref', 15672, 5), ('IncompleteReadError', 2, 1)]
2021-10-16 17:16:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:17:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:17:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('coroutine', 102, 2), ('Task', 43, 1)]
2021-10-16 17:18:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:18:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('coroutine', 103, 1), ('Task', 44, 1), ('TaskStepMethWrapper', 6, 1)]
2021-10-16 17:19:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('frame', 112, 3), ('coroutine', 105, 2)]
2021-10-16 17:19:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:20:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('TaskStepMethWrapper', 7, 1), ('Handle', 35, 1)]
2021-10-16 17:20:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('IncompleteReadError', 3, 1), ('traceback', 6, 1)]
2021-10-16 17:21:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:21:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('coroutine', 109, 4), ('frame', 114, 2), ('Task', 45, 1), ('DNSAddress', 14, 1), ('DNSService', 38, 1)]
2021-10-16 17:22:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('frame', 125, 11), ('traceback', 16, 10), ('coroutine', 111, 2), ('suppress', 6, 2), ('IncompleteReadError', 4, 1)]
2021-10-16 17:22:46 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:23:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:23:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:24:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:24:46 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:25:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:25:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:26:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:26:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:27:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:27:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:28:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('DNSNsec', 15, 1)]
2021-10-16 17:28:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:29:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('DNSNsec', 16, 1)]
2021-10-16 17:29:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('DNSNsec', 19, 3)]
2021-10-16 17:30:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:30:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:31:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:31:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:32:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:32:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:33:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:33:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:34:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:34:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:35:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:35:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:36:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('DNSService', 39, 1)]
2021-10-16 17:36:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('DNSAddress', 16, 2)]
2021-10-16 17:37:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:37:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:38:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:38:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:39:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:39:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('list', 30717, 54), ('dict', 59104, 47), ('QueryMessage', 44, 7), ('Name', 84, 7), ('RRset', 57, 7), ('Handle', 40, 5), ('Task', 48, 3), ('coroutine', 113, 2), ('method', 3230, 1)]
2021-10-16 17:40:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:40:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:41:16 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('coroutine', 123, 10), ('frame', 133, 8), ('Future', 58, 2), ('partial', 205, 2), ('Lock', 81, 1), ('cycle', 9, 1)]
2021-10-16 17:41:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:42:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:42:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:43:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('Future', 59, 1)]
2021-10-16 17:43:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:44:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:44:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:45:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:45:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:46:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('coroutine', 124, 1), ('frame', 134, 1), ('Future', 60, 1), ('FutureIter', 28, 1)]
2021-10-16 17:46:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:47:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:47:46 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:48:16 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:48:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:49:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:49:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:50:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:50:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('weakref', 15674, 2)]
2021-10-16 17:51:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('weakref', 15677, 3)]
2021-10-16 17:51:46 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:52:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:52:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:53:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:53:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:54:16 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:54:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:55:16 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:55:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:56:16 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:56:46 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:57:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('dict_itemiterator', 1, 1)]
2021-10-16 17:57:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:58:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:58:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 17:59:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('CastMessage', 2, 1), ('NullMessageListener', 180, 1), ('_Listener', 403, 1), ('weakproxy', 403, 1)]
2021-10-16 17:59:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:00:16 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:00:46 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:01:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:01:46 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:02:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:02:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:03:16 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:03:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:04:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:04:46 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:05:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:05:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:06:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:06:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:07:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:07:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:08:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:08:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('DNSService', 41, 2)]
2021-10-16 18:09:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:09:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:10:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:10:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:11:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:11:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:12:18 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:12:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:13:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:13:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:14:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:14:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:15:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:15:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:16:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:16:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:17:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:17:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:18:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:18:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:19:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:19:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:20:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:20:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:21:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:21:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:22:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:22:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:23:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:23:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:24:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:24:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:25:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:25:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:26:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:26:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:27:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:27:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:28:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:28:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:29:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:29:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:30:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:30:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:31:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:31:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:32:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:32:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:33:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:33:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:34:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:34:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('ServerDisconnectedError', 2, 1)]
2021-10-16 18:35:18 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:35:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:36:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:36:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:37:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:37:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:38:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:38:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:39:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:39:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('Name', 88, 4), ('QueryMessage', 48, 4), ('RRset', 61, 4)]
2021-10-16 18:40:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:40:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:41:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:41:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:42:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:42:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:43:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:43:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:44:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:44:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:45:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:45:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:46:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:46:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:47:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:47:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:48:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:48:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:49:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:49:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:50:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:50:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:51:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:51:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:52:18 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:52:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:53:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:53:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:54:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:54:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:55:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:55:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:56:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:56:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:57:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:57:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:58:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:58:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 18:59:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('DNSText', 49, 6)]
2021-10-16 18:59:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:00:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('DNSText', 51, 2)]
2021-10-16 19:00:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: [('DNSAddress', 17, 1), ('DNSText', 52, 1)]
2021-10-16 19:01:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:01:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:02:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:02:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:03:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:03:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:04:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:04:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:05:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:05:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:06:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:06:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:07:17 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:07:47 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:08:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: [('defaultdict', 58, 4), ('generator', 15, 3), ('UOWTransaction', 1, 1), ('set', 5154, 1), ('SaveUpdateAll', 1, 1), ('DeleteAll', 1, 1), ('Preprocess', 1, 1), ('SessionTransaction', 2, 1), ('_ConnectionFairy', 1, 1), ('ConnectionEventsDispatch', 3, 1), ('JoinedConnectionEventsDispatch', 1, 1), ('RootTransaction', 1, 1), ('groupby', 1, 1), ('immutabledict', 125, 1), ('SQLiteExecutionContext', 1, 1)]
2021-10-16 19:08:47 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:09:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('function', 73266, 304), ('tuple', 56183, 257), ('cell', 33693, 153), ('PointerType', 487, 134), ('dict', 59202, 98), ('getset_descriptor', 7423, 62), ('FunctionPtrType', 203, 53), ('type', 8520, 33), ('module', 3230, 22), ('ModuleSpec', 3236, 22), ('SourceFileLoader', 3059, 22), ('CType', 316, 11), ('KeyedRef', 84, 7), ('_tuplegetter', 828, 6), ('staticmethod', 1317, 3), ('FileFinder', 498, 3), ('Typedef', 9, 3), ('classmethod', 1087, 2), ('ArrayType', 6, 2), ('_GHash_Imp', 4, 2), ('RawFunctionType', 2, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_ecb.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_cbc.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_cfb.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_ofb.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_ctr.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_Salsa20.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Protocol/_scrypt.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Util/_cpuid_c.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Hash/_ghash_portable.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Hash/_ghash_clmul.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_ocb.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_des.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_des3.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_aes.abi3.so', 1, 1), ('FFILibrary_/usr/local/lib/python3.9/site-packages/Crypto/Util/../Cipher/_raw_aesni.abi3.so', 1, 1), ('Typename', 1, 1)]
2021-10-16 19:09:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: [('tuple', 56197, 14), ('cell', 33702, 9), ('dict', 59205, 3)]
2021-10-16 19:10:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: [('dict', 59211, 6)]
2021-10-16 19:10:47 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:11:17 CRITICAL (SyncWorker_7) [homeassistant.components.profiler] Memory Growth: [('dict', 59227, 16), ('tuple', 56201, 4)]
2021-10-16 19:11:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:12:17 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:12:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:13:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:13:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:14:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:14:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:15:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:15:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('weakref', 15684, 7), ('tuple', 56202, 1)]
2021-10-16 19:16:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('weakref', 15686, 2)]
2021-10-16 19:16:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: [('weakref', 15694, 8)]
2021-10-16 19:17:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: [('weakref', 15707, 13)]
2021-10-16 19:17:47 CRITICAL (SyncWorker_1) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:18:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:18:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:19:17 CRITICAL (SyncWorker_2) [homeassistant.components.profiler] Memory Growth: [('function', 73271, 5)]
2021-10-16 19:19:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:20:17 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:20:47 CRITICAL (SyncWorker_5) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:21:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:21:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:22:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:22:47 CRITICAL (SyncWorker_0) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:23:17 CRITICAL (SyncWorker_4) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:23:47 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:24:17 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:24:47 CRITICAL (SyncWorker_6) [homeassistant.components.profiler] Memory Growth: []
2021-10-16 19:25:17 CRITICAL (SyncWorker_3) [homeassistant.components.profiler] Memory Growth: []RAM usage seems to have stabilized too. I'll keep this going until it crashes, but if it doesn't, it may be that the stream integration was the problem.
bdraco
commented
3 years ago I'm assuming thats with the stream worker (camera)s disabled?
bdraco
commented
3 years ago See profiler results attached... profile.1634423457188610.results.zip
And the Profiler Start log objects... Profiler-Start-log-objects.zip Profiler-Start-log-objects-v1.zip
And visual memory use at time of taking above...
There is very little going on your system based on this.
Its either only going to show up in a py-spy, its not in the python code, or its in another container?.
cad1231
commented
3 years ago I'm assuming thats with the stream worker (camera)s disabled?
That's' right, all camera entities commented out.
gregoinc
commented
3 years ago See profiler results attached... profile.1634423457188610.results.zip And the Profiler Start log objects... Profiler-Start-log-objects.zip Profiler-Start-log-objects-v1.zip And visual memory use at time of taking above...
There is very little going on your system based on this.
Its either only going to show up in a
py-spy, its not in the python code, or its in another container?.
See Py-Spy data attached. Memory still appears to leak, but depends upon what I am adding to the lovelace mostly. py-spy.zip
bdraco
commented
3 years ago See Py-Spy data attached. Memory still appears to leak, but depends upon what I am adding to the lovelace mostly. py-spy.zip
Not much going on in the py-spy as well.
Is the python process using the memory, or is is a sub process?
cad1231
commented
3 years ago @bdraco my system has been working stably without cameras since about 1.30pm yesterday (it is 10.30am right now in my timezone). Previously, it would have crashed at least two or three times in this span of time. There has been no appreciable ram usage growth, it's has been steady around 48% since about 2pm. I think your hunch about cameras may be right. Is there anything else I can do to help pin down the issue further?
jryan82
commented
3 years ago Update here as well, since last update. My system seems stable since 10.5.
I am holding at 9% memory utilization.
cad1231
commented
3 years ago Further update from me. My system has now been stable, without cameras, for about 48 hours. There were a couple of spikes in RAM usage, as you will see in the graph below. I am providing a file with profiler log lines in case something jumps out at @bdraco or someone else.
I will also be rebooting homeassistant with cameras added back in, and will keep running the profiler. I'll post back to see how it goes.
cad1231
commented
3 years ago One thing I noticed that comes up right away when I turned my cameras back on is this error:
2021-10-18 12:52:18 ERROR (stream_worker) [libav.tls] Error in the pull function.
Came up 50 times in about 10 minutes.
I also have a few of these errors:
2021-10-18 12:54:47 ERROR (stream_worker) [homeassistant.components.stream.worker] Error opening stream [uri].m3u8
...and these errors:
Error demuxing stream: [Errno 1414092869] Immediate exit requested: '[uri]m3u8'
...and these errors:
Application provided duration: 8543056563 / timestamp: -461324578552 is out of range for mov/mp4 format
My video streams do sometimes drop out due to the cameras being far away from wifi access points, but hameassistant has handled that gracefully in the past, at least without crashes. Not sure if this is what's causing crashes, will keep an eye on it.
cad1231
commented
3 years ago OK, update. Homeassistant crashed in about 20 minutes when cameras were turned back on. Here is a log of the failures that took place just before the crash.
Clearly something went very wrong in the stream integration after 2010.10. By the way, I updated to 2010.10.6, which is the version that crashed just now.
I should say: throughout this entire time, my cameras were showing just fine in lovelace and were streaming. So I am not sure what is causing these errors.
probot-home-assistant[bot]
commented
3 years ago stream documentation stream source (message by IssueLinks)
probot-home-assistant[bot]
commented
3 years ago Hey there @hunterjm, @uvjustin, @allenporter, mind taking a look at this issue as it has been labeled with an integration (stream) you are listed as a code owner for? Thanks!
(message by CodeOwnersMention)
cad1231
commented
3 years ago I reset it, it worked for about five hours, and now crashed again. Same thing in the logs:
tail home-assistant.log 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 391907004047 / timestamp: 8578808162 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 8519712003 / timestamp: -374808483882 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 400426722078 / timestamp: 8578814192 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 8519712003 / timestamp: -383328195883 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 408946440019 / timestamp: 8578820132 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 8519712003 / timestamp: -391847907884 is out of range for mov/mp4 format 2021-10-18 18:36:40 ERROR (stream_worker) [libav.mp4] Application provided duration: 417466158050 / timestamp: 8578826162 is out of range for mov/mp4 format
gregoinc
commented
3 years ago Interesting to note the discussion around cameras and the stream integration, as I also run cameras. Whilst my issue doesn't appear as extreme as @emjokes I have been taking the approach of restarting the Core after I make any changes to HA. I've found by restarting Core the memory goes back to usual levels and doesn't climb like it had previously. I also found that after installing core-2021.10.5 the problem I was experiencing was less of an issue.
No further action required for me as the situation is manageable, however I would be interested to see what comes from this thread, and wont be installing core-2021.10.6 until I am sure it wont trigger the resurgence of the previous issues.
allenporter
commented
3 years ago There have not been any changes to stream or pyav that I can see in these releases.
@emjokes What type of cameras? I was surprised to see an HLS url in the error log as the stream source.
uvjustin
commented
3 years ago @gregoinc It looks like your usage is spiking at a few key times, and this is different from the slow grind higher that @emjokes is experiencing. I think these are separate issues. We saw something like this before from a misconfigured automation: see https://github.com/home-assistant/core/issues/53267
@allenporter @emjokes
From 2021.09 to 2021.10 we did merge the LL-HLS code, but the functionality there is hidden behind some configuration.yaml options and is disabled by default.
I'm skeptical that this is from the stream component itself. Removing the integration may help since by not using any cameras you are potentially freeing up a hundred MB or more, so there's more free memory as overhead, but that doesn't mean that that the cameras themselves are the cause.
The libav logger messages are very chatty and there will be errors when wifi cameras go offline or reconnect, but these messages should not have changed anytime in a long time since we have not changed the underlying library. Are you sure you did not see them previously?
cad1231
commented
3 years ago @uvjustin @allenporter Thanks for the help. I'm not sure I didn't see these errors in the previous version, in fact, I'm reasonably sure they were there. I'm only pointing them out because they are the last entries in my log before homeassistant crashes.
Interestingly, I did not see a huge spike in ram usage before the crash the last couple of times. RAM use was holding steady, but the core crashed anyway.
As for what kind of cameras, it's four feeds coming from Shinobi, a security monitoring/recording system. Shinobi takes the feed in from the cameras and exposes its own stream url. That's where hls is coming from.
cad1231
commented
3 years ago @uvjustin @allenporter I unchecked "pre-load stream" on two of my camera streams that are not very stable (can drop in and out due to being far from a wifi access point), and the system has been up for about seven hours now, and much more stable.
cad1231
commented
3 years ago Confirming that my system has been stable for well over 24 hours now with "preload stream" UNchecked for two of my cameras. If a developer wants to work with me, I am happy to keep looking for what caused the crashes with logs and performance monitoring tools. I am not a developer myself though, so I can't dig through code. If you search "2021.10 crash", you will see multiple topics about it on hass forums and reddit, so I think it's a somewhat widespread issue.
allenporter
commented
3 years ago @uvjustin ah ok, i do see that 2021.09 was cut August 25th which was before #49608 was merged, but not really seeing how LL HLS related changes could be the problem here (the LL parts are disabled by default, and the non-LL parts seem interesting)
I'm not really sure how to interpret the profiler results other than: It shows 20% of memory used by stream which is normal given that's what its been asked to do by preloading and processing the streams. Agreed that just doing less streaming helps reduce memory pressure.
cad1231
commented
3 years ago @uvjustin ah ok, i do see that 2021.09 was cut August 25th which was before #49608 was merged, but not really seeing how LL HLS related changes could be the problem here (the LL parts are disabled by default, and the non-LL parts seem interesting)
I'm not really sure how to interpret the profiler results other than: It shows 20% of memory used by stream which is normal given that's what its been asked to do by preloading and processing the streams. Agreed that just doing less streaming helps reduce memory pressure.
Trouble is that this was not an issue before 2021.10. Downgrading to 2021.09.7 makes the problem go away entirely, even with preloading streams enabled.
jrbenito
commented
2 years ago Hi there,
I've started to experience this since last upgrade to Version core-2021.11.5 from Version core-2021.11.3 (that I did not see any problem like this).
Proxmox virtual machine with Hassio OVA
Roughly every twelve hours the Core container exists and supervisor does not bring it back online. Therefore I login to the console and issue ha core restart and everything is back online.
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
af684b68f717 ghcr.io/home-assistant/qemux86-64-homeassistant:2021.11.5 "/init" 2 days ago Exited (0) 2 hours ago homeassistant
[...]It was crashed two hours ago and I expected that supervisor should try bring it back up, but not. I took 2 hours to notice because I was out. This morning I also took about two hours to notice because automation lit-up my bedroom in the morning, I went shower, desk was lit up to work (automation) and after some hours I went to kitchen to brew some coffee and no lights nor automation.
Below the two events of today, morning and night. I restarted Core at 9:30 AM more or less this morning and it crashed again around 10PM, roughly twelve hours.
2021-11-25 07:24:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of media_player.spotify_josenivaldo_benito_junior is taking over 10 seconds
2021-11-25 07:24:36 WARNING (MainThread) [homeassistant.components.media_player] Updating spotify media_player took longer than the scheduled update interval 0:00:30
2021-11-25 07:24:40 ERROR (SyncWorker_36) [spotipy.client] Max Retries reached
2021-11-25 07:25:16 WARNING (MainThread) [homeassistant.helpers.entity] Update of media_player.spotify_josenivaldo_benito_junior is taking over 10 seconds
[finish] process exit code 256
[finish] process received signal 9
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
2021-11-25 22:15:47 ERROR (MainThread) [bellows.ezsp] NCP entered failed state. Requesting APP controller restart
2021-11-25 22:15:47 WARNING (MainThread) [homeassistant.components.media_player] Updating androidtv media_player took longer than the scheduled update interval 0:00:10
2021-11-25 22:15:47 WARNING (MainThread) [bellows.zigbee.application] Watchdog heartbeat timeout: EZSP is not running
[finish] process exit code 256
[finish] process received signal 9
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.There are cluster of similar logs as below. Clearly out of memory issue. However, I never had any problem with memory on this PC, I did not increased the number of integrations nor add-ons.
[590946.591038] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=af684b68f717511858d10576be85538136e2acc2157d267699732b75c0ac03bf,mems_allowed=0,global_oom,task_memcg=/docker/e202558f15fc417e55e0d3a59a1d0325af80c68d4560ea919f282566389c0175,task=influxd,pid=504251,uid=0
[590946.591065] Out of memory: Killed process 504251 (influxd) total-vm:4818844kB, anon-rss:424576kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1632kB oom_score_adj:0
[590946.614860] oom_reaper: reaped process 504251 (influxd), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB
[590960.562942] coredns invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
[590960.562945] CPU: 0 PID: 2394 Comm: coredns Not tainted 5.10.75 #1
[590960.562946] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.0.0 02/06/2015
[590960.562948] Call Trace:
[590960.562953] dump_stack+0x6d/0x88
[590960.562955] dump_header+0x45/0x1e8
[590960.562957] oom_kill_process.cold+0xb/0x10
[590960.562959] out_of_memory+0x1a5/0x4b0
[590960.562961] __alloc_pages_slowpath.constprop.0+0xb62/0xc40
[590960.562963] __alloc_pages_nodemask+0x294/0x2f0
[590960.562965] pagecache_get_page+0xb1/0x310
[590960.562966] filemap_fault+0x849/0xc30
[590960.562969] ? __mod_lruvec_state+0x1d/0x40
[590960.562970] ? __mod_memcg_lruvec_state+0x19/0xd0
[590960.562972] ? page_add_file_rmap+0x6f/0xa0
[590960.562974] ? alloc_set_pte+0xec/0x3c0
[590960.562976] ? xas_load+0x5/0x70
[590960.562978] ? xas_find+0x168/0x1b0
[590960.562979] ? filemap_map_pages+0x199/0x3a0
[590960.562981] ext4_filemap_fault+0x28/0x40
[590960.562983] __do_fault+0x32/0x90
[590960.562984] handle_mm_fault+0xb75/0x11c0
[590960.562987] ? ep_poll+0x1e3/0x480
[590960.562989] do_user_addr_fault+0x1b8/0x3f0
[590960.562991] exc_page_fault+0x5a/0x120
[590960.562993] ? asm_exc_page_fault+0x8/0x30
[590960.562994] asm_exc_page_fault+0x1e/0x30
[590960.562996] RIP: 0033:0x4b1a85
[590960.563000] Code: Unable to access opcode bytes at RIP 0x4b1a5b.
[590960.563001] RSP: 002b:000000c0000898f0 EFLAGS: 00010206
[590960.563003] RAX: 0000000000000001 RBX: 000000000043a425 RCX: 00007fe55b0664c8
[590960.563004] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[590960.563005] RBP: 000000c000089910 R08: 00000000108f6601 R09: 0000000000090471
[590960.563005] R10: 00007ffd211d2000 R11: 00029ec6741b3364 R12: 0000000000000003
[590960.563006] R13: 000000c000000780 R14: 0000000000000058 R15: 0000000000000080
[590960.563007] Mem-Info:
[590960.563010] active_anon:35702 inactive_anon:130579 isolated_anon:0
active_file:826 inactive_file:749 isolated_file:32
unevictable:0 dirty:0 writeback:0
slab_reclaimable:8586 slab_unreclaimable:10080
mapped:1295 shmem:2 pagetables:4411 bounce:0
free:5702 free_pcp:510 free_cma:0
[590960.563013] Node 0 active_anon:142892kB inactive_anon:522316kB active_file:3304kB inactive_file:2996kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:5180kB dirty:0kB writeback:0kB shmem:8kB writeback_tmp:0kB kernel_stack:11136kB all_unreclaimable? no
[590960.563013] Node 0 DMA free:7876kB min:40kB low:52kB high:64kB reserved_highatomic:0KB active_anon:528kB inactive_anon:1244kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15004kB managed:10440kB mlocked:0kB pagetables:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[590960.563017] lowmem_reserve[]: 0 1960 1960 1960 1960
[590960.563020] Node 0 DMA32 free:14932kB min:20120kB low:22124kB high:24128kB reserved_highatomic:2048KB active_anon:142112kB inactive_anon:521008kB active_file:3884kB inactive_file:2856kB unevictable:0kB writepending:0kB present:2076928kB managed:966988kB mlocked:0kB pagetables:17632kB bounce:0kB free_pcp:2040kB local_pcp:584kB free_cma:0kB
[590960.563023] lowmem_reserve[]: 0 0 0 0 0
[590960.563026] Node 0 DMA: 5*4kB (UME) 4*8kB (UE) 5*16kB (UME) 2*32kB (UE) 2*64kB (UM) 3*128kB (UME) 2*256kB (UE) 1*512kB (E) 4*1024kB (UME) 1*2048kB (E) 0*4096kB = 7876kB
[590960.563038] Node 0 DMA32: 1186*4kB (UMEH) 607*8kB (UMEH) 203*16kB (UMEH) 57*32kB (UMEH) 11*64kB (H) 1*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 15760kB
[590960.563050] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[590960.563051] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[590960.563052] 1589 total pagecache pages
[590960.563053] 19 pages in swap cache
[590960.563054] Swap cache stats: add 4818429, delete 4818648, find 516570/4791364
[590960.563055] Free swap = 0kB
[590960.563055] Total swap = 506500kB
[590960.563056] 522983 pages RAM
[590960.563057] 0 pages HighMem/MovableOnly
[590960.563057] 278626 pages reserved
[...]
[590960.563242] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=e460fe7d263db4a3a885b6147335eed76956094e7140a104bac10b7ae87732c9,mems_allowed=0,global_oom,task_memcg=/docker/af684b68f717511858d10576be85538136e2acc2157d267699732b75c0ac03bf,task=python3,pid=464720,uid=0
[590960.563297] Out of memory: Killed process 464720 (python3) total-vm:565920kB, anon-rss:155864kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1144kB oom_score_adj:0
[590960.628770] oom_reaper: reaped process 464720 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kBBefore last upgrade the swap was pretty stable. Out of nowhere it becomes very active since yesterday:
Usage also increased to max it out:
I realize that I have only 2 GiB of RAM allocated and that is because in the past I used it on a RPi3 with just 1GiB. Since I never experienced any issue with those 2GiB I just forgot to come back and increase it. I just added one more GiB to see how it goes.
If I can collect any more information, please let me know.
BRs, José
The problem
Home asssitant crashes after update to 2021.10 crashes every 12 hours or so. No changes have been made to the configuration from the previous version
What is version of Home Assistant Core has the issue?
core-2021.10.2
What was the last working version of Home Assistant Core?
core-2021.9.7
What type of installation are you running?
Home Assistant OS
Integration causing the issue
No response
Link to integration documentation on our website
No response
Example YAML snippet
Anything in the logs that might be useful for us?
No response
Additional information
No response from the user interface or via ssh, I am forced to reboot the virtual machine from proxmox.