Occasionally high memory usage

nazar-pc commented 1 month ago

The problem

Originally reported in https://github.com/home-assistant/operating-system/issues/3483

I believe this is the third time this happened to me. Home Assistant is usually running fine for days with stable memory usage and then suddenly memory usage is growing significantly. Once it hanged the whole thing, two more times (including today) just resulted in high memory usage and monitoring alerts: Знімок екрана з 2024-07-22 12-57-18

I looked inside and python3 -m homeassistant --config /config is the process that is currently using 56.6% of RAM (on VM with just 2G of RAM).

Memory usage started to recover for some reason, have not done anything to cause that :man_shrugging: Home Assistant is not reachable from public Internet.

There is no records in Home Assistant Core logs. Supervisor logs have this, looks like it restarted today, though no idea why it would do that (doesn't correlate with the memory usage increase either):

2024-07-22 09:34:07.792 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
2024-07-22 09:34:07.793 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
2024-07-22 09:35:04.575 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
2024-07-22 09:35:16.031 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
2024-07-22 09:35:18.259 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/loganmarchione/hassos-addons repository
2024-07-22 09:35:18.271 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/music-assistant/home-assistant-addon repository
2024-07-22 09:35:18.277 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/home-assistant-addon repository
2024-07-22 09:35:18.282 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/addons repository
2024-07-22 09:35:18.289 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository

Nothing interesting in any other logs.

What version of Home Assistant Core has the issue?

2024.7.0

What was the last working version of Home Assistant Core?

No response

What type of installation are you running?

Home Assistant OS

Integration causing the issue

No response

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

Eventually it crashed Home Assistant, things restarted and memory usage is back to normal: Знімок екрана з 2024-07-22 21-03-03

joostlek commented 1 month ago

Include a callgrind file from the https://www.home-assistant.io/integrations/profiler/ profiler.start service
Include a py-spy from https://community.home-assistant.io/t/instructions-to-install-py-spy-on-haos/480473. (be sure to zip them up before posting as github mutates svg files)
If memory is leaking, include 4 hours of RAW logs (Settings -> System -> Logs -> Download) from the https://www.home-assistant.io/integrations/profiler/ profiler.start_log_objects service with the default settings

home-assistant[bot] commented 1 month ago

Hey there @bdraco, mind taking a look at this issue as it has been labeled with an integration (profiler) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `profiler` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign profiler` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

profiler documentation profiler source _{^{(message by IssueLinks)}}

issue-triage-workflows[bot] commented 4 weeks ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

nazar-pc commented 4 weeks ago

I'm not sure it was clear from my initial message, but this happens suddenly and very rarely. I can't simply start profiler and wait for it to happen :shrug:

bdraco commented 4 weeks ago

I'm not sure it was clear from my initial message, but this happens suddenly and very rarely. I can't simply start profiler and wait for it to happen 🤷

Sadly, that's the worst kind of problem to track down. You could automate the monitoring of memory usage and start the profiler with an automation that watches the system monitor. You may get lucky and catch it.

nazar-pc commented 4 weeks ago

I do have Prometheus client running on HA instance if that is what you mean, though not sure how would I automatically start profiler based on those metrics.

I guess what I'm looking for is more or less exact steps what to do and what kind of information to save in case it happens again in the future. It usually continues to work with high memory usage for some time before crashing and restarting.

Wouldn't say this is a massive issue for me, just hoping to collect useful information for maintainers since something somewhere does go wrong in those cases.

nazar-pc commented 3 weeks ago

Just happened again, I had to reboot it before it crashed. Here are the logs before reboot, I have not seen anything particularly interesting: home-assistant_2024-08-16T14-29-11.733Z.log

issue-triage-workflows[bot] commented 1 week ago

There hasn't been any activity on this issue recently. Due to the high number of incoming GitHub notifications, we have to clean some of the old issues, as many of them have already been resolved with the latest updates. Please make sure to update to the latest Home Assistant version and check if that solves the issue. Let us know if that works for you by adding a comment 👍 This issue has now been marked as stale and will be closed if no further activity occurs. Thank you for your contributions.

home-assistant / core