demo.openwebrtc.org doesn't work between bowser ans safari with plugin

[truestep]

1. I started Safari with Webrtc plugin installed and daemon running on Mac, like the doc demands.
2. On iPad I launched bowser. Created and joined a session.
3. On Safari on Mac opened the link proposed and Called. Result:
4. On Mac with Safari with plugin I can see local video. Remote video is black rectangle.
5. On iPad's bowser I can see local video. Remote video is black rectangle. (tried a few times)

In console output of the daemon I see: (:9256): GStreamer-CRITICAL **: Trying to dispose element source-output-queue-2, but it is in PLAYING instead of the NULL state. You need to explicitly set elements to the NULL state before dropping the final reference, to allow them to clean up. This problem may also be caused by a refcounting bug in the application or some element.

Also when I use my app from bowser to my app in another bowser remotestreamadded event is not fired at all, While between chrome and bowser it's ok and can be used somehow.

[truestep]

Bowser to bowser from ipad to ipad on demo.openwebrtc.org also doesn't work. I can't see the remote video at all. Even no black rectangle.

[stefanalund]

What specific devices are you using? I tested this same scenario 2 days ago on a iPhone 6s and an iPad Air 1st, it worked.

[truestep]

iPad 3 (iOS 9.1) mini and iPad 2 (iOS 9.1) And Safari with plugin to iPad 3 (iOS 9.1)

[stefanalund]

@stefhak have you tried openwebrtc-daemon to Bowser recently?

[stefhak]

Just tried, and it did not work very well :( If openwebrt-daemoninitiates the error log (in openwebrt-daemon) looks like:

0:00:42.396557000 53885 0x7fb4b36020c0 ERROR             owrsession owr_session.c:767:_owr_session_emit_ice_state_changed:<OwrMediaSession@0x7fb4b403eba0> Session 1, ICE failed to establish a connection!
ICE state changed from gathering to failed
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtcp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtcp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtcp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====

and there is only one-way video (to Bowser). If Bowser initiates the error log (in openwebrt-daemon) is:

0:00:26.358477000 53889 0x7fb65a51f4c0 ERROR             owrsession owr_session.c:767:_owr_session_emit_ice_state_changed:<OwrMediaSession@0x7fb65b04cba0> Session 1, ICE failed to establish a connection!
ICE state changed from disconnected to failed

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:53889): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:53889): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)
==== Error message start ====
Error in element srtp-encoder.
Error: GStreamer encountered a general supporting library error.
Debugging info: gstsrtpenc.c(1058): gst_srtp_enc_process_buffer (): /GstPipeline:transport-agent-1/GstBin:transport_bin_1/GstBin:send-output-bin-1/GstDtlsSrtpEnc:dtls_srtp_rtp_encoder_1/GstSrtpEnc:srtp-encoder:
Unable to protect buffer (protect failed) code 2
==== Error message stop ====

and there is remote video in Bowser only for a short while before it stops.

However, this is not isolated to Bowser - OWR, OWR - OWR (using Safari and daemon) also gives only one-way video, and a similar log in (in openwebrt-daemon):

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)

(<unknown>:57990): libnice-WARNING **: Could not find component 2 in stream 1

** (<unknown>:57990): WARNING **: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (cands_added > 0)
0:00:38.312380000 57990 0x7faab9601cc0 ERROR             owrsession owr_session.c:767:_owr_session_emit_ice_state_changed:<OwrMediaSession@0x7faaba052660> Session 1, ICE failed to establish a connection!
ICE state changed from gathering to failed

[superdump]

This is being investigated.

[alessandrod]

* (:57990): WARNING _: (owr_transport_agent.c:2414):on_new_remote_candidate: runtime check failed: (candsadded > 0) (:57990): libnice-WARNING *: Could not find component 2 in stream 1

I have been looking at this one. I can reliably reproduce it when doing FFOX <-> OWR and Chrome <-> OWR and initiating from FFOX and Chrome. I never get it when initiating with OWR.

It's happening because both FFOX and Chrome specify rtcp-mux in the offer SDP, but then send candidates for both RTP and RTCP. This is the correct behaviour according to the standard. According to the standard in fact when initiating a call with rtcp-mux the caller must assume that the receiver doesn't support the rtcp-mux extension (for backward compatibility) and so still send separate candidates for the RTP and RTCP components. Then if the receiver does support rtcp-mux, it's supposed to ignore remote RTCP candidates and only reply with local candidates for muxed RTP/RTCP.

We get the warning because initially when we see rtcp-mux in an offer SDP we configure the nice streams to expect only one component. Then we get remote candidates for two components tho (RTP and RTCP) and when we add those to the agent, the agent is expecting only once component. The warning is effectively innocuous. We just need to better handle this nuance of the rtcp-mux extension.

[superdump]

OK, something like if rtcp-mux and component 2, discard candidate?

[alessandrod]

OK, something like if rtcp-mux and component 2, discard candidate?

Yep exactly

[pererikb]

There is a check already: https://github.com/EricssonResearch/openwebrtc/blob/master/bridge/worker/peerhandler.js#L309 but if fails since the bindings are broken after the addition of OWR_COMPONENT_MAX

[superdump]

@pererikb - that's fine for the bridge. I'm not sure how reasonable it is to expect every application to implement discarding the candidates for the RTCP component. Would it not be reasonable to implement this behind the C API? Given that it is incorrect behaviour to add those candidates anyway.

[pererikb]

I think warnings are appropriate for programming errors, we should not silently discard candidates that are incorrectly added.

[alessandrod]

I think at the very least the warning should be made more explicit. It took a debugger, reading the libnice code and reading the rtcp-mux RFC to understand what was going on.

[superdump]

I think requiring significant knowledge of WebRTC without making it super clear where to find that knowledge is very un-user-friendly.

"Candidate added for disabled component - rtcp-mux has been negotiated so candidates for RTCP must not be added."

It sounds like an irritation for a user to me. Why not drop the candidate and warn that we're doing it because it isn't a programming error, it's helping the user to adhere to a requirement according to specifications?

[superdump]

Or maybe owr_session_add_remote_candidate should return some error state to indicate what happened and why.

[pererikb]

There's already a more specific warning: https://github.com/EricssonResearch/openwebrtc/blob/master/transport/owr_session.c#L384 We just have to make sure that it's actually triggered.

[superdump]

Indeed!