Open jamjambles opened 1 year ago
Interested in pointers to potential solutions. My use case is similar, generate some text-to-speech and play it back.
The example you mentioned (https://github.com/whitphx/streamlit-webrtc/blob/main/pages/8_media_files_streaming.py) uses a callback to process the video frames (video_frame_callback
).
Does using audio_frame_callback
in this place instead work for you?
The audio filter example may also be a reference about the usage of audio callback while it's a client-to-server example.
Interested in pointers to potential solutions. My use case is similar, generate some text-to-speech and play it back.
@wenshutang I'm also interested in this issue for my text-to-speech streaming problem. I wonder if you've found a solution yet. If so, could you please share some references or suggestions? Thank you.
Many of the examples in this repo show client to server media sinks (mic / video capture), which have frame based callback processing. I am looking to do server to client media playback, with frame based callback processing. This would be useful for real-time audio playback with real-time processing.
After searching through this discussion https://discuss.streamlit.io/t/new-component-streamlit-webrtc-a-new-way-to-deal-with-real-time-media-streams/8669, and the example pages in streamlit-webrtc, I have not been able to find an example of this.
To be specific, I am looking to do the following:
This example uses the MediaPlayer class from aiortc: https://github.com/whitphx/streamlit-webrtc/blob/ff697dc0fb85df58dec2307251e57105ebb737bb/pages/8_media_files_streaming.py#L9. However it does not seem that this provides any sort of callback on the stream (at the audio frame level).
Digging deeper, the MediaPlayer class has a MediaStreamTrack instance (https://aiortc.readthedocs.io/en/latest/api.html#aiortc.MediaStreamTrack) which has a
recv
callback method for each frame.Would the correct approach be to create a new subclass of MediaStreamTrack and write a custom
recv
for the required processing? I found this related thread: https://github.com/aiortc/aiortc/issues/571Is this functionality supported currently? I would appreciate any guidance here.
Thanks heaps!