Connection Reliability Issues

[mlamoure]

I've been noticing an occasional seemingly random web socket error that does not recover until I restart the plugin. I had no network issues while this was happening. But, I found a corresponding error message in the HA logs that seems to indicate there's a flood of messages that the plugin or the HA socket integration are unable to keep pace with, all shared below.

Here's the Indigo log when that happens:

Mar 21, 2024 at 5:44:36 PM
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: socket is already closed.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.
   Home Assistant Agent Error      Websocket error: Connection to remote host was lost.

I went to the plugin log for that timestamp, here's what I found:

2024-03-21 17:44:35.535 [THREADDEBUG]               Plugin.threaddebug              Ignoring update from entity `sensor.delta_pro_1_status`, no matching Indigo device found
2024-03-21 17:44:36.534 [   ERROR]               Plugin.on_error                 Websocket error: Connection to remote host was lost.
2024-03-21 17:44:36.537 [   DEBUG]               Plugin.start_websocket          start_websocket called with delay=5, using 10.66.0.87:8123

I'm happy to send you a more complete plugin log via email. I'd rather not post it here.

Here's the HA log error message at the same timestamp, with small redactions for privacy:

[140055585772608] Mike from 10.66.0.3: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"opensky_exit","data":{"callsign":"JZA434","altitude":null,"sensor":"OpenSky","longitude":-xxx,"latitude":xxx,"icao24":"c055eb"},"origin":"LOCAL","time_fired":"2024-03-21T21:44:36.508799+00:00","context":{"id":"01HSHFJ6WWHF2D633NS9N0K1PY","parent_id":null,"user_id":null}},"id":32}'
[140055589300544] Mike from 10.66.0.3: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"opensky_exit","data":{"callsign":"JZA434","altitude":null,"sensor":"OpenSky","longitude":-xxx,"latitude":xxx,"icao24":"c055eb"},"origin":"LOCAL","time_fired":"2024-03-21T21:44:36.508799+00:00","context":{"id":"01HSHFJ6WWHF2D633NS9N0K1PY","parent_id":null,"user_id":null}},"id":34}'
[140055605243712] Mike from 10.66.0.3: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"opensky_exit","data":{"callsign":"JZA434","altitude":null,"sensor":"OpenSky","longitude":-xxx,"latitude":xxx,"icao24":"c055eb"},"origin":"LOCAL","time_fired":"2024-03-21T21:44:36.508799+00:00","context":{"id":"01HSHFJ6WWHF2D633NS9N0K1PY","parent_id":null,"user_id":null}},"id":36}'
[140055837913280] Mike from 10.66.0.3: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"opensky_exit","data":{"callsign":"JZA434","altitude":null,"sensor":"OpenSky","longitude":-xxx,"latitude":xxx,"icao24":"c055eb"},"origin":"LOCAL","time_fired":"2024-03-21T21:44:36.508799+00:00","context":{"id":"01HSHFJ6WWHF2D633NS9N0K1PY","parent_id":null,"user_id":null}},"id":38}'
[140055575626816] Mike from 10.66.0.3: Client unable to keep up with pending messages. Reached 4096 pending messages. The system's load is too high or an integration is misbehaving; Last message was: b'{"type":"event","event":{"event_type":"opensky_exit","data":{"callsign":"JZA434","altitude":null,"sensor":"OpenSky","longitude":-xxx,"latitude":xxx,"icao24":"c055eb"},"origin":"LOCAL","time_fired":"2024-03-21T21:44:36.508799+00:00","context":{"id":"01HSHFJ6WWHF2D633NS9N0K1PY","parent_id":null,"user_id":null}},"id":40}'

[FlyingDiver]

I'm thinking this may be related to the comment in your thread by another user who said that significant HA traffic spiked the Indigo server CPU.

But looking at those error messages, it looks like your OpenSky integration is flooding the connection with updates. That excerpt shows 5 event messages, all with the same timestamp.

I would disable that integration and see if the problem goes away.

[mlamoure]

Ok turned off and will monitor. I wasn't using that integration anyway.

[FlyingDiver]

Reopen if you see it again. At that point I'll need some data on the frequency of updates of the HA entities. I might need to streamline the message processing to reduce the overhead.