The system's load is too high or an integration is misbehaving

[Mariusthvdb]

as of Home Assistant 2023.3.0b we are seeing this in the add-on logs:

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/http.py:186 
Integration: Home Assistant WebSocket API (documentation, issues) 
First occurred: 06:56:37 (1 occurrences) 
Last logged: 06:56:37

[140507648324032] AppDaemon from 172.30.33.4: Client unable to keep up with pending messages. Stayed over 1024 for 5 seconds. The system's load is too high or an integration is misbehaving

and, new to this version, we can see it registering to all services in HA:

2023-02-27 14:36:57.210397 INFO HASS: Registering new service script/buttons_set_hue_outdoors_sensitivity
2023-02-27 14:36:57.210576 INFO HASS: Registering new service script/set_hue_alarm_time_wakeup
2023-02-27 14:36:57.210765 INFO HASS: Registering new service script/reload
2023-02-27 14:36:57.210945 INFO HASS: Registering new service script/turn_on
2023-02-27 14:36:57.211132 INFO HASS: Registering new service script/turn_off
2023-02-27 14:36:57.211330 INFO HASS: Registering new service script/toggle
2023-02-27 14:36:57.211622 INFO HASS: Registering new service device_tracker/see
2023-02-27 14:36:57.211850 INFO HASS: Registering new service unifi/reconnect_client
2023-02-27 14:36:57.212039 INFO HASS: Registering new service unifi/remove_clients
etcetc

consider the above to be a small snippet.

Ive reported this in the beta channel on Discord, and Paulus replies

issue with AppDaemon not being able to keep up is not an HA issue and needs to be fixed by AppDaemon

Ive linked to your community post https://community.home-assistant.io/t/announcement-appdaemon-4-2-3/541010 and get notified that if Appdeamon

are using the new API in a release, the add-on will have to be updated to that release too. However, the release you linked mentions explicitly that there is no ARM support

so, in a bit if a catch22 now...

please have a look, and let me know what to do.

[acockburn]

What hardware are you running on, and what version of AD? (not the add-on version). If you are running on 4.2.1, that hasn't changed in a year so I don't see how this can be an AppDaemon issue - at a minimum it is a change in behavior in HASS that is causing AD issues. I am not up to date with the new API - what is it doing? Is it a breaking change?

Also, there is still support for ARM, we didn't change that, we were just unable to create the Docker Hub multi arch build with ARM images. Manual builds continue to work fine,

I'm not aware of any changes in 4.2.2 that would affect this area either, so I'll upgrade HASS when the new version comes out and see what happens.

[acockburn]

Also, I just checked with @frenck and the add-on is built with a custom docker images so is not in any way affected by the removal of ARM support from the docker hub images.

[Mariusthvdb]

yeah, I was surprised as well, but have not deeper knowledge than what I passed along above.

my hardware is a MIniPC and has been for 2 years now. so no changes there. the AD version I wouldn't know how to tell, all I can say is I am using HA OS 2023.3.0b and indeed the community add-on (by Frenck)

https://github.com/hassio-addons/addon-appdaemon/blob/main/appdaemon/requirements.txt states 4.2.1 as you already suspected

[balloob]

For listening to entity updates, use subscribe_entities instead of listening to all events. It's highly compressed and only sends deltas. Listening to all events is way too much data, as each state changed event includes fully both old + new state.

[acockburn]

Thanks for the heads up Paulus, I'll look into this. I'm not convinced that this is a general problem for AD users as nothing significant seems to have changed in either HASS or AD, but it would make a very nice optimization, albeit its a huge change to the way AD works.

[bdraco]

The relevant change in 2023.3.x beta is that we now report the name of the client and some more details about the client. Previously we received quite a few of these reports but it was non obvious about the source which lead to the improved logging.

[acockburn]

Ok well after a quick look at the API I don’t think I have what I need to make the proposed change yet - AD makes available all HASS events and that is important for a lot of applications. I understand the potential in optimizing the state changes as I am sure they are by far the largest piece, and I can write code to subscribe to specific entities that users are listening to no problem, but I would need a way to also sub to all events that aren’t state changes - I don’t see an easy way to do that although I could be wrong, I’m not aware of a definitive list of events, and there doesn’t seem to be a filtering option that would allow this although I’ll keep digging. I might need to get some changes into HASS to support this which is a job in itself. Happy to be proved wrong on this piece as I like the idea of the changes - would appreciate a second opinion after my hasty reading of the WS commands module in the hass code.

[bdraco]

The current frontend design uses subscribe_entities and than subscribes to a list of specific events. If you need everything that's always going to be a bit slow.

There are two wins that may help quite a bit in that case:

1. Switch the json parser to orjson
2. Enable the message coalesce feature on the web socket api

[bdraco]

These might help

https://github.com/home-assistant/home-assistant-js-websocket/pull/306

https://github.com/zachowj/node-red-contrib-home-assistant-websocket/commit/d235c3572ec5051cf11bc3bf26c9266f5ca9c659

[bdraco]

Longer term I think we would have to add a way to subscribe to all events except state changes if there is no way around needing all events.

We currently effectively only have an include filter but not an exclude filter option afaict

[acockburn]

Great information thanks - I’ll put some thought into maybe requiring the user to list the events they care about - they should know. And I’ll look into those other suggestions too. I don’t see this as a huge problem or I would have heard about it from other users but the WS API has evolved significantly since I wrote AD and I should make changes to benefit - I’m actually still using REST calls which are no longer shown in the docs so I assume they have been deprecated- for that reason alone I should probably rewrite the AD code that integrates with HASS.

[acockburn]

That would be desirable, but now I think about it, it shouldn’t be too much of a stretch to require my users to list the events they need. I’ll think about it some more but that will be my working assumption, since most users won’t need anything but state changes. I’ll ask around and see what the users think.

[acockburn]

And as a further thought, I could have AD automate the process, so it would be pretty seamless in fact. If they care about an event and listen for it, AD will subscribe to it in hass automatically.

[acockburn]

@Mariusthvdb How big is your system in terms of numbers of devices/sensors etc.? I am adding some perf measurements into AD so we can get a better idea of what is going on, and my system has a fair number of sensors, but is peaking at less than 10K/s as a 10 second average. Most of the time it is at zero. I'm still interested in building a more efficient plugin for HASS, but I don't see this as a huge pending problem for the average user, and I am curious what setup you have that is overrunning AD, or if perhaps the latest version has some different characteristics. When I release the next version can you have a look at the admin screen plugin section to let me know what kind of numbers you are seeing? It's available now in the dev build if you are able to use that, if not we can wait until the next version is released.

[Mariusthvdb]

just t give you an idea (couldn't get it all in 1 screenshot):

and HA keeps creating new entities from attributes ;-) without kidding, it has been quite large since I started out with AD some years ago, and havent touched those apps really. They are really tiny btw, and only take care of some basic logging on binary motion sensors. So their own workload couldn't be it.....

I'll try and have a go add that add-on, but not sure I ever did that for add-ons, run a non release version.

[acockburn]

OK, I will relate a small version in the next few days to test our new CI pipeline, so you will get it then anyway.

[github-actions[bot]]

This issue is stale because it has been open for 45 days with no activity. Remove stale label or comment or this will be closed in 15 days.

[github-actions[bot]]

Please feel free re-open this issue if you have new information available