Assist sometimes fails to recognise end of speech

HarvsG commented 1 month ago

The problem

During my experiments over the year of the voice and since, I have hit upon a common issue. When making voice commands to assist it has often failed to detect the end of speech, and so the command does not execute until the listening phase times out, giving the impression of a slow-to-respond voice assistant. But the voice assistant debug page shows that each component responds quickly. Search terms: Assist slow to respond, 15 second delay before response. I and others have documented this issue elsewhere:

https://github.com/rhasspy/wyoming-satellite/issues/20, <- occurred with satellite and assist microphone
Forum post 1
Forum post 2
Discord thread where we made some progress after having the same issue with an M5 Stack Atom Echo.

But it seems that the code responsible for end of speech detection is part of core: components/assist_pipeline/vad.py and not the satellites or devices as I had thought. Although I expect there might be some variation in future about where end of speech detection happens.

Given my success tweaking some ESPHome config settings in ameliorating (but not completely resolving) the issue, it seems that the quality/volume/gain of the audio being fed in is one of the biggest determining factors, so it might not be an issue with Core per-se, but I think this is the best place to discuss it and this is where the processing happens and is the common denominator across several devices and platforms.

It seems there are a number of possible ways this could be improved in my mind:

Detection and logging of this error within HA
- Adding one of these for a voice detection phase in the VA debug panel would be a good first step
- Throw a warning log here: #122182 https://github.com/home-assistant/core/blob/de5b5f6d360ecbc981db4f69043631e81d594703/homeassistant/components/assist_pipeline/vad.py#L161-L163
Silero VAD was reported to be helpful by some here https://github.com/rhasspy/wyoming-satellite/issues/20
End of speech detection is done elsewhere, such as on device
Better documentation and tools to ensure that the audio streams are pre-amplified and filtered appropriately

What version of Home Assistant Core has the issue?

2024.7.2

What was the last working version of Home Assistant Core?

NA

What type of installation are you running?

Home Assistant OS

Integration causing the issue

assist_pipeline

Link to integration documentation on our website

https://www.home-assistant.io/integrations/assist_pipeline/

Diagnostics information

See linked issues and thread for diagnostics if they are useful

Example YAML snippet

No response

Anything in the logs that might be useful for us?

No response

Additional information

No response

home-assistant[bot] commented 1 month ago

Hey there @balloob, @synesthesiam, mind taking a look at this issue as it has been labeled with an integration (assist_pipeline) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of `assist_pipeline` can trigger bot actions by commenting: - `@home-assistant close` Closes the issue. - `@home-assistant rename Awesome new title` Renames the issue. - `@home-assistant reopen` Reopen the issue. - `@home-assistant unassign assist_pipeline` Removes the current integration label and assignees on the issue, add the integration domain after the command. - `@home-assistant add-label needs-more-information` Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue. - `@home-assistant remove-label needs-more-information` Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

assist_pipeline documentation assist_pipeline source _{^{(message by IssueLinks)}}

HarvsG commented 1 month ago

Switch from WebRTC to microVAD #122861 May improve this issue.

home-assistant / core