search

home-assistant / core

:house_with_garden: Open source home automation that puts local control and privacy first.
https://www.home-assistant.io
Apache License 2.0
71.63k stars 29.93k forks source link

Core OOM killed during DST #58791

Closed basschipper closed 2 years ago

basschipper commented 2 years ago

The problem

This morning I discovered that my HA Core was down after the DST change (NLD located). From the dmesg log I conclude that HA core was OOM killed. After reviewing the home assistant log it seems that two of my automations, that also start at exactly 2 o'clock, seem to have stalled? Home Assistant is continually complaining that these two automations are "Already running". This repeats many times until the OOM killer kills Home Assistant.

Bringing HA Core back is just matter of typing: ha core start.

Which also reaches the question why the supervisor didn't restart HA Core?

What happend to these automations during DST? Maybe it's better to run time based automations at 5 minutes past the hour?

What version of Home Assistant Core has the issue?

core-2021.10.6

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Automation

Link to integration documentation on our website

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Dmesg:

[665397.236593] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=b9a094bf125883ccd4620ef09874aa2a7338bca139ac8231e3a4d220eef70454,mems_allowed=0,global_oom,task_memcg=/docker/b9a094bf125883ccd4620ef09874aa2a7338bca139ac8231e3a4d220eef70454,task=python3,pid=2142414,uid=0
[665397.236650] Out of memory: Killed process 2142414 (python3) total-vm:4974216kB, anon-rss:3375328kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:9664kB oom_score_adj:0

HA Core:
2021-10-31 02:51:47 INFO (MainThread) [buienradar.buienradar_json] Parse ws data: latitude: ***, longitude: ***
2021-10-31 02:51:47 INFO (MainThread) [buienradar.buienradar_json] Parse ws data: latitude: ***, longitude: ***
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Running automation actions
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Executing step call service
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Running automation actions
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Executing step call service
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Running automation actions
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Executing step call service
... OTGW and Luftdaten logs repeating many times ...
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Running automation actions
2021-10-31 02:00:00 INFO (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Executing step call service
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:00 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
... OTGW and Luftdaten logs repeating many times ...
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
[finish] process exit code 256
[finish] process received signal 9
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
[s6-finish] sending all processes the TERM signal.
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.luftdaten_push] Luftdaten Push: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
2021-10-31 02:00:05 WARNING (MainThread) [homeassistant.components.automation.otgw_outside_temperature] OTGW Outside Temperature: Already running
... OTGW and Luftdaten logs repeating many times ...
[s6-finish] sending all processes the KILL signal and exiting.

Additional information

No response

hmmbob commented 2 years ago

Yeah, duplicate of https://github.com/home-assistant/core/issues/58783

basschipper commented 2 years ago

Moved my issue to #58783.