Commands execute, feedback delayed, logs full of `Error executing action: Call timed out`

[rohankapoorcom]

I've been using companion (various versions) with Home Assistant for ~4 years now (also various versions). In the past few weeks, I've noticed connection issues with Home Assistant and Companion that are disrupting this functionality.

Usually, making a change in an entity in Home Assistant is reflected (via the Feedback) in Companion instantly. This is no longer happening. Usually, pushing a companion button triggers an instant entity change in Home Assistant (or script call) and then the feedback updates instantly as well. However, this is also not happening. The entity change in Home Assistant does happen instantly, but the feedback update is often delayed by 20-40 seconds (if it even happens at all). This makes it very difficult to use for something like a microphone mute button since you don't know whether you are muted or not.

I've tested this with every version of Home Assistant from 2023.8 onwards until 2023.12 (I've not upgraded beyond that yet) along with every combination of Companion versions 3.1.2, 3.2.0, 3.2.1 and all of them exhibit the same behavior. The problem began while I was running Home Assistant 2023.8 and companion 3.1.2 (which had previously been working fine for months). During that time, I had added more automations and entities to Home Assistant.

I've also tried factory resetting Companion and then importing my configuration and the problem persists.

Log file attached: 3c64966b835b_companion_log_20240210-1249.csv

Hopefully this is enough to figure out the issue or come up with more troubleshooting steps.

Environment

- OS: Ubuntu 22.04 (Docker Host), Windows 10 / Mac OS X Sonoma (both clients)
- Browser: Chrome 120.0.6099.201
- Companion Version: 3.2.1 (3.2.1+6676-stable-fa981d25)
- Home Assistant Version: 2023.12.4

[drewancil]

Not sure how helpful this is, but I also had trouble start about this time. I have a large Home Assistant install, and just connecting Companion to HA will cause the companion device to lock up in 15-20 minutes. I used Companion successfully before this. I thought that maybe the device was overloaded (Raspberry Pi 4b/8GB), so I used an Intel NUC with 16GB of memory and an SSD and it locked that up too.

Ive switched up Companion to use MQTT to interface with HA which is much less convenience and powerful, but it works. Is this integration still maintained?

[rohankapoorcom]

I've noticed the locking up as well. Companion itself seems fine, but all the integrations in Companion stop working.

How many entities do you have in Home Assistant? I've got about 5300 and I'm wondering if the entity count is part of the issue.

Can you share more details about how you're controlling entities via MQTT?

Tagging @Julusian in case you're able to help. I believe you're the maintainer of this plug-in.

[drewancil]

I'm at 6510 entities today. Yea, I wondered the same as you about size and this issue. I figured if it was universal there would be more reports of issues.

The strange thing is that with a clean install, just enabling the HA integration in Companion causes the lockup - I don't even have to call any of the functions, use feedback, anything. I was surprised it would use resources in this way. I'm pretty handy with Python, but know zero about JS/TS, so I haven't even looked at the code.

For MQTT, I just use the generic MQTT integration in Companion to send an MQTT message to a topic with an associated payload on the same MQTT broker I use for Home Assistant (I'm using the add-on). Then I just create automations that trigger on various MQTT topics and run actions based on the payloads. Not nearly as easy/seamless as Companion, and you don't get feedback, but for most of what I was doing with Companion, it ended up being pretty easy and I didn't need the feedback. But, it was fast and everything was very stable under this method. I'd much like to return to using the HA integration as it worked great for me before the timeframe you mentioned you started having trouble.

I'm here to troubleshoot or test anything, but unfortunately I'm not able to provide the time or expertise to fix at code level.

[rohankapoorcom]

I tried updating to 3.2.2 (3.2.2+6688-stable-7417d2a0) today to see if it would make a difference. It behaves exactly the same as before.

The strange thing is that with a clean install, just enabling the HA integration in Companion causes the lockup

Did you try a clean install of HA or of Companion only? I tried spinning up a minimal home assistant instance with no integrations configured and Companion did not lock up at all. So I believe there is a strong correlation with entity count and the instability that we are both seeing. I'm wondering if Companion is trying to track the state of every entity in HA rather than only tracking the ones that are used for buttons or feedback.

Thanks for sharing those details with your MQTT setup. Unfortunately for me I have about 30 entities / scripts that I control in Home Assistant through Companion and need the Feedback. I'm considering trying this alternative software instead, but I'm hopeful that this module in Companion can be fixed since I do use other functionality inside Companion directly.

[drewancil]

Yea, I periodically load the latest beta of Companion and test - locks up in 15-20 minutes or so.

The clean instance I set up was Companion to my existing HA. I did not try a clean HA, but I suspect my findings would equal yours.

Did you watch attributes like memory, cpu, and disk i/o when you enable? Does it exhaust any of those resources?

It would be lovely to hear from the author - even if they do not plan to look into so we can make other arrangements if so. It would be a shame because Companion works very well otherwise.

[rohankapoorcom]

Did you watch attributes like memory, cpu, and disk i/o when you enable? Does it exhaust any of those resources?

On Companion? No, nothing appears to be exhausted. Nor on Home Assistant either since all of the clients are able to continue using it with no disruptions.

I had a crazy idea of setting up a second Home Assistant instance for just Companion use and having it handle a small subset of entities from the primary one, but that doesn't seem like a great idea from the maintainability perspective.

[rohankapoorcom]

@drewancil I haven't tested this yet, but found this integration for Home Assistant that may be helpful. It looks like it will let you publish a copy of an entity over MQTT and have it automatically configured for discovery and interaction with a second Home Assistant instance. It seems like it would be a reasonable way to take the ~30 or so integrations that I care about interacting with via companion and put them on a dedicated Home Assistant instance for Companion and see if that stabilizes the problem.

It's not an ideal solution, but it seems like it could be the best workaround for me. I'll report back once I have some time to test it out (hopefully this weekend).

[rohankapoorcom]

I just finished setting up the integration mentioned in my previous message to publish specific entities (~45) to a new Home Assistant integration via MQTT. Pointing Companion to that instance of Home Assistant (with only those entities on it) resulted in a perfectly responsive and functional Companion setup.

I think this confirms that something is going awry with Companion when connected to Home Assistant instances with thousands of entities.

[prankousky]

Is there any initiative to fix this problem? I just set up companion on a raspberry pi just to control Home Assistant. While it works fine, connection will drop quite soon, likely due to the mentioned huge amount of entities.

While the mqttstream and second HA is a nice idea, it'd be even better if it would still work out of the box for those of us with too many ;) entities.