whitphx / streamlit-webrtc

Real-time video and audio streams over the network, with Streamlit.
https://discuss.streamlit.io/t/new-component-streamlit-webrtc-a-new-way-to-deal-with-real-time-media-streams/8669
MIT License
1.28k stars 178 forks source link

Add frame callbacks with SENDONLY mode #1069

Closed whitphx closed 11 months ago

whitphx commented 1 year ago

Make it possible to inject the callbacks between the input tracks and the receivers.

This helps such cases where video-only input and audio-only output are needed like https://discuss.streamlit.io/t/new-component-streamlit-webrtc-a-new-way-to-deal-with-real-time-media-streams/8669/155

To do it, the following code should work while it is not working as the frame callbacks are not supported with SENDONLY mode.

import logging

from streamlit_webrtc import webrtc_streamer, WebRtcMode
import av

logger = logging.getLogger(__name__)

def video_frame_callback(frame: av.VideoFrame) -> av.VideoFrame:
    ...
    return frame

def audio_frame_callback(frame: av.AudioFrame) -> av.AudioFrame:
    ...
    return frame

ctx = webrtc_streamer(
    key="video_input",
    mode=WebRtcMode.SENDONLY,
    video_frame_callback=video_frame_callback,
    audio_frame_callback=audio_frame_callback,
    media_stream_constraints={"video": True, "audio": False},
)

webrtc_streamer(
    key="audio_output",
    mode=WebRtcMode.RECVONLY,
    video_frame_callback=video_frame_callback,
    media_stream_constraints={"video": False, "audio": True},
    source_audio_track=ctx.output_audio_track,
    desired_playing_state=ctx.state.playing
)
gustavz commented 1 year ago

any update on this?