ayufan / camera-streamer

High-performance low-latency camera streamer for Raspberry PI's
293 stars 46 forks source link

Adding trickle ICE support to speed up connection speeds and fully support the WebRTC protocol #68

Closed QuinnDamerell closed 1 year ago

QuinnDamerell commented 1 year ago

Since libdatachannels is advertising support for trickle ICE in the SDP offer, we should support it since WebRTC clients will use it. Tickle ICE lets the connection be faster by streaming the possible ICE candidates out as they become available instead of waiting until all have been resolved and then sending the remote description.

This change also allows all ICE candidates to be considered for possible candidates for the connection. Before this, only one ICE candidate was returned in the answer SDK; thus, it was the only one "agreed upon" by the client and server. This worked in most local setups since there was only one server candidate anyways. But even in local scenarios where the device has more than one IP address (like v4 and v6), ICE negotiation could fail. This change also enables lower-priority ICE candidates like STUN or TURN servers to be used as fallbacks.

QuinnDamerell commented 1 year ago

@ayufan, note that I also added the signaling_protocol_version for compatibility. That way, frontends can figure out which version of camera-streamer they are talking to and can thus know if the remote_candidate request is valid or they need to wait to gather all of the candidates before sending the SDP answer.

Please feel free to change and/or modify anything you wish!

QuinnDamerell commented 1 year ago

@ayufan it can be done by waiting, but tickle ice is normally preferred and faster, because the clients can connect as soon as a single ice candidate is sent.

https://webrtc.org/getting-started/peer-connections#trickle_ice

The problem with gathering is you must wait synchronously during the entire gathering process. If there's a slow turn server or something, it can hold up the entire process. But if the turn sever isn't even needed, there's no need to wait on it.

ayufan commented 1 year ago

@QuinnDamerell I don't mind adding endpoint, but I think we can cut on complexity:

  1. Send all know peers in a request that were received up-to this point.
  2. Once request was send, send additional icecandidate via remote_candidate.

This allows to:

  1. Remove std::vector<rtc::Candidate> pending_remote_candidates; and possibly bool has_set_sdp_answer = false;.
  2. Make http_webrtc_remote_candidate to always do addRemoteCandidate or error out.
  3. Make http_webrtc_request to fetch current ice candidates from json body.

WDYT about this?

QuinnDamerell commented 1 year ago

@ayufan that would work in theory, but I think you will always need pending_remote_candidates. In practice, WebRTC will always give you the local answer SDP almost instantly before any ICE candidates have been resolved. The ICE candidates can start coming in quickly after it, but there's a bit of a gap. But pending_remote_candidates is needed because even though you will always get the answer SDP before a candidate, it's not always true that the request to put the answer SDP will beat the ICE request due to possible network issues. And thus, since you can't add the ice candidate before the SDP answer, I think the list will always be needed. That or you can require clients to implement it correctly, where they can't send ice candidates until the SDP answer request is done. But IMO it's better to make the API easier for clients.

QuinnDamerell commented 1 year ago

I was thinking it might be helpful to make the remote_candidate take a list of candidates, so a client could send multiple bached together if it wanted to.

ayufan commented 1 year ago

@QuinnDamerell I will play with it tomorrow.

QuinnDamerell commented 1 year ago

@ayufan no problem, please let me know if I can help at all!

QuinnDamerell commented 1 year ago

@ayufan took on this work and added it in a slightly different way!