search

whitphx / streamlit-webrtc

Real-time video and audio streams over the network, with Streamlit.
https://discuss.streamlit.io/t/new-component-streamlit-webrtc-a-new-way-to-deal-with-real-time-media-streams/8669
MIT License
1.27k stars 176 forks source link

Audio recording callback for ICE connection state SUCCESS #470

Open oplatek opened 2 years ago

oplatek commented 2 years ago

Hi there!

Thank you for the streamlit-webrtc. I was able to create a recording demo using the gist mentioned in the issue.

I noticed that there is a latency before clicking the "Start" button and establishing the connection. In the logs I see

2021-10-20 16:26:39.128 Connection(0) Check CandidatePair(('10.0.38.38', 33812) -> ('172.20.10.2', 63080)) State.FROZEN -> State.WAITING
2021-10-20 16:26:39.128 ICE connection state is checking              
2021-10-20 16:26:39.469 Connection(0) Check CandidatePair(('10.0.38.38', 33812) -> ('172.20.10.2', 63080)) State.WAITING -> State.IN_PROGRESS
2021-10-20 16:26:40.210 Connection(0) Discovered peer reflexive candidate Candidate(Tte2XxBsnN 1 udp 1853824767 46.135.1.113 18117 typ prflx)
2021-10-20 16:26:40.210 Connection(0) Check CandidatePair(('10.0.38.38', 33812) -> ('46.135.1.113', 18117)) State.WAITING -> State.IN_PROGRESS
2021-10-20 16:26:40.329 Connection(0) Check CandidatePair(('10.0.38.38', 33812) -> ('46.135.1.113', 18117)) State.IN_PROGRESS -> State.SUCCEEDED
2021-10-20 16:26:40.330 Connection(0) ICE completed
2021-10-20 16:26:40.332 ICE connection state is completed             
2021-10-20 16:26:52.721 Track audio ended
2021-10-20 16:27:24.035 Connection(0) Consent to send expired         
2021-10-20 16:27:24.035 ICE connection state is failed
2021-10-20 16:27:24.036 ICE connection state is closed                
2021-10-20 16:28:02.403 Track audio received

I would like to have a callback if the Connection state changes to and from State.SUCCEEDED, so I can inform the user that he/she is recorded or the connection stopped (or is not initialized yet) and he is not recorded.

Is it possible to check the state by polling or register a callback when in changes?

whitphx commented 2 years ago

Please try polling ctx.state.signalling and ctx.state.playing, while I'm not sure whether this completely meets your demand. These statuses will change as below.

  1. Before clicking "start": (signalling, playing) = (False, False)
  2. After clicking "start": (signalling, playing) = (True, False)
  3. After the streaming starts: (signalling, playing) = (False, True)

However, callbacks are not supported now, so you have to do polling these statuses with while-loop.

Regarding callbacks, I will ask the Streamlit team if it is possible to implement that mechanism in a custom component. -> https://discuss.streamlit.io/t/a-callback-with-custom-components/18388