search

dummylabs / thewatchman

Home Assistant custom integration to keep track of missing entities and actions in your config files
MIT License
475 stars 20 forks source link

Watchman Causes HA to Crash or Become Temporarily Unresponsive #67

Open DanGarion opened 1 year ago

DanGarion commented 1 year ago

Home Assistant 2022.12.0 Supervisor 2022.11.2 Frontend 20221207.0 - latest Watchman v0.6.0

I've been working on tracking down an issue I'm having when I update, edit, or delete an Automation, Script, or Scene in my HA installation and it has lead me to believe that Watchman is causing my HA to become temporarily unstable when when it updates.

I see messages like this in my logs whenever I save a change to an automation, script, or scene. Dec 08 14:42:03 elsa homeassistant[914]: 2022-12-08 14:42:03.423 INFO (MainThread) [custom_components.watchman] 1968 files parsed and 0 files ignored in 34.59s. due to HA restart

Typically what happens is I edit an automation, hit save and my dashboard becomes unresponsive. After 35-45 seconds I see the message above in my system logs and the dashboard either starts to respond or my HA appears as though it restarts.

All the times today I have noticed this happen are when watchman updated. image

bcutter commented 1 year ago

I can confirm this behavior. Must be the way Watchman is integrated or updates itself.

All the actions listed (reloading automations, scripts and scenes) seem to trigger reloading Watchman too. So far so good.

BUT as this blocks the UI for quite some time (roughly 20 to 25 seconds on my system) for all endpoints, this is quite a serious issue. So in my case HA becomes unresponsive. I don't see "crashes" (yet).

I installed Watchman just 15 minutes ago and discovered this bad behavior and I'm really thinking about removing this integration just because of this. Having such a negative impact on the overall system caused by an integration is a no-go.

I might additionally note:

bcutter commented 1 year ago

@dummylabs Is it (meanwhile/as interims solution) possible to stop Watchman from updating when reloading automations, scripts and scenes?

HA core team put quite some effort in speeding up reloading automations etc. including only reloading single automations/scripts/scenes this year. All of that is colpotized by Watchman... 😞

Update: https://www.home-assistant.io/blog/2022/11/02/release-202211/#smarter-reloading-of-automation--scripts

rjgrandy commented 12 months ago

I also see this behavior and finally discovered watchman as the culprit, was driving me insane....

bcutter commented 12 months ago

Indeed. An integration causing Core to get fully blocked (for 30 seconds in my case, your mileage may vary due to system performance) is an absolute no-go.

The biggest no-go is, this issue is very old - and nobody fixed it until today. What a pity.

infestonn commented 8 months ago

Took me weeks to identify what had been causing to freeze my HA when I click reload "All yaml configuration". Without Watchman it usually takes 20s, with Watchmap – 2 minutes.

bcutter commented 8 months ago

Even more annoying is it intercepts editing automations and scripts ⚠️⚠️⚠️

On every save (using the UI, which probably causes scripts.yaml and automations.yaml to be parsed again bei WM) my HA is frozen for roughly 30 seconds. Caused by Watchman.

Not sure if this integration is maintained at all cause such a major issue not being worked on for years ⚠️ is very sad.

ildar170975 commented 8 months ago

causing to freeze my HA when I click reload "All yaml configuration"

Same as "why a PC antivirus makes my file operations slower???". What else do you expect? Do not run Watchman constantly. Watchman is processing config changes as it is supposed.

bcutter commented 8 months ago

Do not run Watchman constantly.

Do you really mean this like you wrote it? Is this a "You are holding it wrong!" blaming? Are we now discussing if this issue even is an issue? In case we do I think this is the final dead plug for this integration :-)

ildar170975 commented 8 months ago

Is this a "You are holding it wrong!" blaming?

You are exaggerating, dear disliker. And yes, this is about an intelligent use of Watchman. If it tracks EVERY config_change & re-evaluate itself - then possible (not mutually excepting) solutions could be: -- do not react in every config change since they could be like an avalanche; -- do not run it constantly, run it periodically.

dummylabs commented 2 days ago

v0.6.3 should be fully compliant with HA async requirements for custom integrations

bcutter commented 2 days ago

v0.6.3 should be fully compliant with HA async requirements for custom integrations

...unfortunately saving automations or scripts in the UI still makes whole HA freeze for 20 to 30 seconds when Watchman is enabled.

Or do you mean 0.6.4 (unreleased yet) and you tested the new version?

dummylabs commented 2 days ago

No, nothing has changed in version 0.6.4 according to this issue yet. Ok, then we need a switch in the Watchman settings to disable it. Just out of curiosity, what hardware do you run Home Assistant on?

bcutter commented 2 days ago

That switch would be nice. It would allow things like using the automation_reload event to trigger an automation which waits for 30 (or with Watchman disabled only 1 to 3) seconds and enables Watchman (again) etc.

In short: good idea 👍

My HA runs on a Pi 4.