Initiator answer fails when responder adds streams

[xdumaine]

This might be something I'm doing wrong, but I haven't been able to track down a reason.

I'm getting the error offerer must use actpass value for setup attribute from

TraceablePeerConnection.createOffer at
PeerConnection.setLocalDescription at
PeerConnection.CreateAnswer at
PeerConnection.answer at
PeerConnection.handleOffer at
JingleMediaSession.onSourceAdd

When I start the session, this all fires off as normal, but when the responder adds a stream, the error is thrown by when the answer occurs on self.pc.answer in onSourceAdd in jingle-media-session on L491. The new stream from the responder does actually come through, and can be displayed by the initiator, but the answer fails to process with the error above.

A couple oddities that I'm seeing, and I'm not sure if they're related:

1. The Initiator gets a "default" stream in peerStreamAdded event that seems to have no media actually tied to it - is this expected? I can't find where this is coming from.
2. No peerStreamAdded event is fired on the responder's session, even though the initiator created the session with createMediaSession(jid, sid, stream) with a valid stream. The event still doesn't occur even if the initiator manually calls addStream on the session.

Any help you're able to lend would be greatly appreciated. If there's a better forum to discuss this than the issues, please let me know!

[legastero]

Looks like the mid-session addStream() method needs to take into account whether you are the initiator or not. Otherwise, the DTLS client/server roles can be wrong depending on who adds the stream and produce these errors.

This will happen for P2P sessions, which explains why we've not hit this before since when using a video bridge all of the web clients always act as DTLS clients.

[fippo]

https://code.google.com/p/webrtc/issues/detail?id=2782 -- that is the chrome bug for this

[xdumaine]

I could avoid doing the mid-session addStream call (which I'll probably have to do for firefox anyway, right?), if the initial stream came through properly. Do you have any ideas why the responder would not be getting any stream from the initiator?

[xdumaine]

I think I may have found the problem with the initiator's stream- jingle.createMediaSession accepts stream as it's third argument, but it never gets added to the session. I have to explicitly call session.addStream(stream) after creating the session, and before starting it. If that sounds right, I'd propose a simple if (stream) { session.addStream(stream) } check and call added createMediaSession or if (opts.stream) { this.addStream(opts.stream); } to jingle-media-session in the constructor. If that sounds right to you, I'll open a PR.

[sarumjanuch]

I just started working with Jingle in stanza.io. And did faced same issue, when i got to the code, i was not able to realize how it supposed to be working(using cached local\remote desc for renigotiation), i believe, that this may work with jistsi video bridge but not peer to peer, i was able to add streams in the middle of the session only after i modified media session code: created handler for onnegotiationneeded, and inside that handler i create new offer, send it to remote party, also i modified onSourceAdd, to clearly handle an offer, crate answer, and send it back to the offerer. Any time you add\remove stream you need to do true renegotiate, not just setting cached local\remote description. So in p2p model i was able to modify streams only with such scenario. Any comments?

[xdumaine]

@sarumjanuch I'd love to see your code. Could you open branch/PR?

[sarumjanuch]

@xdumaine will do, right now i am trying to clean what i did, and testing everything (addStream, removeStream, switchStream).

[sarumjanuch]

I made a very unclean commit just to show you what i am doing, so you have some clue, https://github.com/sarumjanuch/jingle-media-session/commit/65bd67be6178693f87fb3aa81f0476b1ecf8573f, also to test this you need to comment out: https://github.com/otalk/jingle.js/blob/master/index.js#L308, i know it is very unclean now. This allows me to add\remove streams during the call between two chrome browsers. I have separate streams for audio and video. so when i start an audio call i call getUserMedia with audio: true, video:false, when i want to do video call, i call getUserMedia twice one with audio: true, video:false and second with audio: false, video:true, same for answer. I can answer with audio only or with A\V, also i can add/remove streams during the call (addStream3) so when you started audio only call both call parties can add\remove video\screensharing stream during the call. Now the sad story this is not working with Firefox :( It don't have the pc.removeStream... I am thinking that i can disable\enable video stream once it added instead of adding removing it every time, or try to replace tracks, but this is not the biggest issue. for example when i start audio call from Chrome to FF with OfferToReceiveVideo = true; and on firefox i answer without video, i am getting errors on chrome that it was not able to push down SDP, and many others... So if anyone familiar with chrome to FF interop i would be really glad to talk with.

[xdumaine]

@sarumjanuch The idea of using multiple streams is definitely clever, and looks like you got it to work with Chrome, so that's cool. For Firefox, looks like you're running into a few issues. The first is that Firefox removed removeStream and uses removeTrack instead (which is, among other reasons, to give the finer grain control and prevent having to use multiple stream objects). The second might be that Firefox no longer uses OfferToReceiveVideo but instead offerToReceiveVideo (note the case difference). The third potential problem I see is the old "unified vs plan b" issue with sdp.

[sarumjanuch]

Maybe anyone know? why FF is creating local SDP that neither FF || Crome can't push down as remote desc: v=0 o=mozilla...THIS_IS_SDPARTA-43.0.4 4335663485117234844 0 IN IP4 0.0.0.0 s=- t=0 0 a=sendrecv a=fingerprint:sha-256 94:EC:6D:64:9D:AE:7A:FE:90:E6:08:81:0A:FD:BD:D3:AE:C7:66:4E:72:D9:23:68:24:EC:13:BB:EE:C5:F5:7E a=group:BUNDLE audio a=ice-options:trickle a=msid-semantic:WMS * m=audio 33798 RTP/SAVPF 111 c=IN IP4 188.230.44.10 a=candidate:0 1 UDP 2122252543 IP 33798 typ host a=candidate:2 1 UDP 2122187007 IP 60841 typ host a=candidate:1 1 UDP 1686052863 IP 33798 typ srflx raddr IP rport 33798 a=candidate:3 1 UDP 1685987327 IP 60841 typ srflx raddr IP rport 60841 a=sendrecv a=end-of-candidates a=extmap:1/sendrecv urn:ietf:params:rtp-hdrext:ssrc-audio-level a=ice-pwd:2cd8e7e3c4bca99d1df2230e8a123625 a=ice-ufrag:228519e4 a=mid:audio a=msid:{badc7ae7-71df-4e31-822f-0ad1ea100cbf} {f114182b-e21f-4a53-8347-c752ca26ad04} a=rtcp-mux a=rtpmap:111 opus/48000/2 a=setup:active a=ssrc:2997826411 cname:{fca05d23-35ad-43f1-b8f6-380075241ead} m=video 0 RTP/SAVPF 0 c=IN IP4 0.0.0.0 a=inactive a=rtpmap:0 PCMU/8000

this: m=video 0 RTP/SAVPF 0 c=IN IP4 0.0.0.0 a=inactive a=rtpmap:0 PCMU/8000

have on ice ufrag,pwd.. how it supposed to receive video on it, if remote party will enable video?

[sarumjanuch]

Failed to set remote answer sdp: Called with SDP without ice-ufrag and ice-pwd.

[fippo]

that m-line has been rejected by firefox. Chrome should not care about the lack of an ice-ufrag/pwd here. Can you grab a chrome://webrtc-internals log?

[sarumjanuch]

Sure i can, here is the log for the chrome calling Firefox and firefox sends an answer, which neither chrome neither firefox can handle, they both telling that no ice-ufrag and ice-pwd: http://pastebin.com/sWCgua9G,

[fippo]

ah... the problem is that the m-line is rejected by firefox but jingle can't really convey that -- we haven't had time or the usecase to implement not sending back the video content while preserving the m-line to make sdp happy

[sarumjanuch]

Philipp, can you explain me a little bit more, on what needs to be done, to resolve an issue, i already have some clue in yours library internals, so i can implement what you will tell me. I would be immensely gratefull, i you would give me an advise, what i should do next to fix this.

[sarumjanuch]

Right now i came up with: var sdp = SdpParser.parse(answer.sdp); sdp.media.forEach(function(media){ if(media.type === 'video' && media.inactive ) { delete media.fingerprint; media.port = 0; } }); answer.sdp = SdpParser.format(sdp); That fixed an issue. Now i have another.. after firefox adds stream, and after renegotiation chrome want to add stream it can't set localDescription: Failed to set local offer sdp: Session error code: ERROR_CONTENT. Session error description: Failed to set local video description recv parameters..

[fippo]

the problem is that it's not clear how to do it at the protocol level / doing the right thing is a lot of extra effort.

I'd go for "no transport with ufrag/pwd" instead of looking for media.inactive

[sarumjanuch]

I was able to fix: "Failed to set local offer sdp: Session error code: ERROR_CONTENT. Session error description: Failed to set local video description recv parameters.." By doing: if(this.is === chrome) { offer.sdp = offer.sdp.replace(/a=rtpmap:\d+ rtx\/\d+\r\n/i, ""); offer.sdp = offer.sdp.replace(/a=fmtp:\d+ apt=\d+\r\n/i, ""); } Now i have only one issue left with interop Chrome & firefox. 1) Chrome calls Firefox with A\V (two streams from chrome) 2) Firefox Answers only with audio 3) Here everything is ok, chrome have audio FF have A\V 3) Firefox adds video stream 4) Chrome receives an offer, but have black screen for incoming video from FF and report packet loss for video rcv stream... All other scenarios is working by now. I am thinking this somehow related to ICE, but can't find out what is wrong. Philipp can you look in to this trace, maybe you will see something that is wrong: http://pastebin.com/gZm8JEiS

[sarumjanuch]

This is what i see in ff debug log: All could not pair new trickle candidate, i have already tried dfferent timeouts for this.wtFirefox, but this doesn't help :(

[sarumjanuch]

I fixed the above also, there is some bug in chrome or FF(i think it's chrome), where they can't renegotiate when there is 100 an 120 payloads in offer, so simply remove 120 payload if 100 is present.. Now i did find new one, and i am already going to give up on that idea with streams, and with jingle. When i call from FF to Chrome after session established and everything is on place, and i want to add stream on Chrome, for some reason chrome changes ice-ufrag, ice-pwd in offer(Ice restart or what?), which leads to inability for firefox to push-down remote decryption as it don't support ice restarts...

Any ideas why Chome might want to restart ice after adding stream and renegotiation, when initial offerer was FF?

C to C and F to F, C to F don't have such behavior.. only when call starting as F to C.

[jensengar]

Thanks to @sarumjanuch I was able to find a solution to p2p stream surgery. In our application, we wanted the ability to actually turn off the camera when video was muted rather than just mute the stream/track. Our video and audio is on the same stream and it was easy to stop the video track and rip it out on mute. The tricky thing was adding video back as we were running into the same problems as mentioned above. @sarumjanuch said that the problem was with using a cached version of the remote description in switchStream(). The solution I took may be somewhat hacky but it's fairly straight forward and minimal code is required. In a nutshell, since our p2p connection always have a datachannel, I decided to do a full-blown offer/answer through the data channel. Here are my steps and I'll include some code later:

When we need to add video back to the call (Peer 1)

1. create new stream with audio and video
2. stop old stream
3. session.removeStream(oldStream)
4. session.addStream(newStream)
5. createOffer
6. setLocalDescription
7. send offer over datachannel

(Peer 2)

7. setRemoteDescription from datachannel message
8. createAnswer
9. setLocalDescription
10. send answer over datachannel

(Peer 1)

11. setRemoteDescription
12. magic happens.

The hacky-ness comes in because we have to bypass the jingle-media-session decoration. See the following:

(steps 1-6)

session.removeStream(session.pc.getLocalStreams()[0]);
session.addStream(newStream);
session.pc.pc.createOffer((offer) => {
    session.pc.pc.setLocalDescription(offer);
    // send offer over datachannel
});

(the rest)

handleSessionMessage(session, data) {
    session.pc.pc.setRemoteDescription(new RTCSessionDescription(data.payload));
    if (data.type === 'dataChannelOffer') {
        session.pc.pc.createAnswer((answer) => {
            session.pc.pc.setLocalDescription(answer);
            // send answer back
        });
    }
}

If you see any problems with what I've done or there are concerns, please let me know

[sarumjanuch]

Hi there! Yes, that is the basic logic i have started with. But I am using separate stream for each track. And we do also offer screen sharing, so call might have up to 3 streams (when Chrome to chrome and FF to FF). Tricky part is when chrome and FF trying to talk with each other. For example when ff calls to chrome, chrome answers without video and then in mid call chrome wants to add video, at this moment, chrome decides to do ICE restart (as initial ff offer was with sdparta_x unifiedPlan, and new offer by chrome will be now with mid=audio\video PlanB, and new ICE credentials), which is not supported by FF, so this will fail. And many other nasty bugs. My solution was to create a logic where chrome never knows anything about sdparta_x and also hacking on ssl role in sdp, and some payloads changes. Also i have created a logic that is detecting local and remote browser on call start, to have any clue when and what hacks needs to be applied.

[jensengar]

Ahh yes. I didn't check the interop. Can you elaborate a bit more on what you did here?

My solution was to create a logic where chrome never knows anything about sdparta_x and also hacking on ssl role in sdp, and some payloads changes.

And why does this only happen on reconnects? I obviously have no problem setting up the initial connection between ff and chrome, why would it be different mid call?

[sarumjanuch]

Well something like this:

            //Workaround for an issue https://bugs.chromium.org/p/webrtc/issues/detail?id=3962
            if(adapter.webrtcDetectedBrowser === 'chrome') {
                offer.sdp = offer.sdp.replace(/a=rtpmap:\d+ rtx\/\d+\r\n/i, "");
                offer.sdp = offer.sdp.replace(/a=fmtp:\d+ apt=\d+\r\n/i, "");
            }
            //Fixes an issue when call between Chrome and Firefox. and firefox offers 100 and 120 Payloads
            //https://bugs.chromium.org/p/webrtc/issues/detail?id=5450#
            if(offer.sdp.indexOf('a=rtpmap:120 VP8/90000') >= 0 && offer.sdp.indexOf('a=rtpmap:100 VP8/90000') >= 0){
                offer.sdp = offer.sdp.replace(/a=rtpmap:120 VP8\/90000\r\n/i, "");
                offer.sdp = offer.sdp.replace(/a=rtcp-fb:120 nack\r\n/i,"");
                offer.sdp = offer.sdp.replace(/a=rtcp-fb:120 nack pli\r\n/i,"");
                offer.sdp = offer.sdp.replace(/a=rtcp-fb:120 ccm fir\r\n/i,"");
            }
            self.pc.setLocalDescription(offer,
                function () {
                    var jingle;
                    if (self.config.useJingle) {
                        //To PlanB
                        if(adapter.webrtcDetectedBrowser === 'firefox' && self.isInitiator && self.calleeBrowser === 'chrome') {
                            offer.sdp = offer.sdp.replace(/a=mid:sdparta_1\r\n/i, "a=mid:video\r\n");
                            offer.sdp = offer.sdp.replace(/a=mid:sdparta_0\r\n/i, "a=mid:audio\r\n");
                            offer.sdp = offer.sdp.replace(/a=group:BUNDLE sdparta_0 sdparta_1\r\n/i, "a=group:BUNDLE audio video\r\n");
                        }

    //Fixes an issue when Firefox swaps the DTLS role, and chrome then reporting ssl role conflict error.
    if(adapter.webrtcDetectedBrowser === 'chrome' && !self.isInitiator && self.callerBrowser === 'firefox') {
        answer.sdp = answer.sdp.replace(/a=setup:active\r\n/i, "a=setup:passive\r\n");
    }

[sarumjanuch]

But my implementation have a limitation max 2 streams between chrome and ff. so when i need screen sharing and already have a video, instead of adding additional stream, i am switching it.

[sarumjanuch]

Oh yes, one more awesome thing:

        this.pc.addStream(stream);
        if(navigator.mozGetUserMedia) {
            setTimeout(function(){
                self.emit('NegotiationNeeded');
            },500);
        }
        this.once('NegotiationNeeded',function(){

Super cool world of WebRTC. They should rename it to HackRTC.

[jensengar]

Yeah that's pretty rough. Thanks for figuring all that out though! This gives me a lot to go on.

[medabida]

hey, any help will be appreciated, im using stanza.io + ejabberd and whenever i try to establish a call between Firefox or chrome , i get this error: Failed to set remote answer sdp: Called with SDP without ice-ufrag and ice-pwd (everything is OK between 2 chrome browsers) ive already posted a question on stackoverflow: https://stackoverflow.com/questions/47740113/webrtc-could-not-process-webrtc-answer

[sarumjanuch]

This thread already has an answer to your question.

[medabida]

@sarumjanuch well im looking for an explanation more than a solution, when accepting a call with audio and video stream , everything seems to be perfect (across all browsers). my only problem (at least for the moment) is when i accept a call that contain audio only, the browser throws Failed to set remote answer sdp: Called with SDP without ice-ufrag and ice-pwd this is the sdp (when call is audio only) v=0\r\no=- 1513356029841 1513356030735 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=msid-semantic: WMS *\r\na=group:BUNDLE sdparta_0\r\nm=audio 1 UDP/TLS/RTP/SAVPF 109 101\r\nc=IN IP4 0.0.0.0\r\na=rtcp:1 IN IP4 0.0.0.0\r\na=ice-ufrag:9f446a57\r\na=ice-pwd:bfd597c964a14889c893c72edc50db59\r\na=fingerprint:sha-256 F4:37:DE:0A:CA:76:47:6B:3B:3C:87:BB:72:3E:B8:AB:E7:14:82:90:11:67:82:D7:DC:22:E8:B9:51:53:D3:9B\r\na=setup:active\r\na=sendrecv\r\na=mid:sdparta_0\r\na=msid:{53ada2f7-2562-4e1d-8c13-93567338ae11} {5f4235e3-dcb2-4c75-abb8-b2d37ae5044d}\r\na=rtcp-mux\r\na=rtpmap:109 opus/48000/2\r\na=fmtp:109 maxplaybackrate=48000;stereo=1;useinbandfec=1\r\na=rtpmap:101 telephone-event/8000\r\na=fmtp:101 0-15\r\na=ssrc:1015361223 cname:{413e0664-1243-4195-899c-7e413d2eb670}\r\na=ssrc:1015361223 msid:{53ada2f7-2562-4e1d-8c13-93567338ae11} {5f4235e3-dcb2-4c75-abb8-b2d37ae5044d}\r\nm=video 1 UDP/TLS/RTP/SAVPF 120\r\nc=IN IP4 0.0.0.0\r\na=rtcp:1 IN IP4 0.0.0.0\r\na=fingerprint:sha-256 F4:37:DE:0A:CA:76:47:6B:3B:3C:87:BB:72:3E:B8:AB:E7:14:82:90:11:67:82:D7:DC:22:E8:B9:51:53:D3:9B\r\na=inactive\r\na=mid:sdparta_1\r\na=rtpmap:120 VP8/90000\r\n how can it contain the video mlines ?

[sarumjanuch]

@fippo told: > ah... the problem is that the m-line is rejected by firefox but jingle can't really convey that -- we haven't had time or the usecase to implement not sending back the video content while preserving the m-line to make sdp happy

[medabida]

so should i write my own code in order to make it work (basically tweaking the jingle js file to make it conform to the audio only session like described in this url ? or is there is a workaround that i can do ?

[sarumjanuch]

so should i write my own code in order to make it work (basically tweaking the jingle js

Yes.

[medabida]

well i solved the problem by including the sdpparser in the client js (of stanza io) then building it again, and in stanza.io.bundle ive tweaked the PeerConnection.prototype.handleAnswer to remove the media fingerprint of the video if call is audio only. thanks for the help