search

csdcorp / speech_to_text

A Flutter plugin that exposes device specific text to speech recognition capability.
BSD 3-Clause "New" or "Revised" License
348 stars 217 forks source link

Audio Turned Off After Using STT with Flutter WebRTC #479

Closed emrecodeway closed 3 months ago

emrecodeway commented 3 months ago

Description

I am developing a Flutter application where I use both the speech_to_text package for speech recognition and flutter_webrtc for video streaming. The application functions correctly with the WebRTC video streaming, including audio, until speech recognition is initiated. After using the startListening (and startListeningiOS for iOS) method from the speech_to_text package, the speech is successfully recognized and processed. However, once the STT session is stopped (using stopListeningiOS), the audio from the WebRTC stream is turned off, and I'm unable to re-enable it.

Steps to Reproduce

  1. Start the application with WebRTC streaming, audio and video are working fine.
  2. Initiate STT session with startListening/startListeningiOS.
  3. Fetch the speech text and process it.
  4. Stop the STT session with stopListeningiOS.
  5. The WebRTC video continues to play, but the audio is turned off.

Expected Behavior

After the STT session is stopped, the audio from the WebRTC stream should continue to play as normal.

Actual Behavior

The audio from the WebRTC stream is turned off after the STT session is completed and does not resume.

Additional Information

It seems like the STT functionality might be interfering with the WebRTC audio stream, and there doesn't appear to be a re-initialization or reactivation of the audio stream after the STT session is completed. I am looking for guidance on how to manage the WebRTC audio stream state in conjunction with the STT functionalities or if there's a known issue/workaround for this scenario.

Any assistance or insight into this issue would be greatly appreciated.

sowens-csd commented 3 months ago

I want to make sure that we're talking about the same thing. There is no startListening or startListeningiOS method in the STT package. The STT method is listen There is a startListening method in the example so perhaps you've called your methods the same?

For WebRTC, and other sound packages that use sound, there's no way to share the underlying sound resources of the OS, at least none that I've found. So you have to shutdown the WebRTC session while doing speech recognition then start it back up again afterwards.

emrecodeway commented 3 months ago

Thank you for your response and clarification.

I would like to confirm that startListening and startListeningiOS are indeed custom methods in my implementation, inspired by the example code provided in the package documentation. These methods function as expected.

However, the core issue I'm encountering is with the flutter_webrtc audio stream. Specifically, the audio gets muted after invoking the listen method from the package. This muting persists even after I execute the stop and cancel methods of the speech recognition package, before forwarding the recognized speech results to the streaming service.

On your suggestion, I will attempt to close the WebRTC session and then start a new one post-STT interaction, and I will observe that this approach does indeed resolve the audio issue. It seems like managing the WebRTC session state around the STT interactions is a viable workaround.

Thank you again for your assistance. It has been instrumental in helping me identify a solution.