onaddstream never called

[pantaoran]

I've been debugging this for days now but can't figure it out. We are building a server that does some video processing on the video that clients send to it via WebRTC.

After the SDP offer and answer are exchanged and the setRemoteDescription was called on the client (Bowser), I expect an onaddstream event so that I can attach the remote stream to my video element, but that event is never called. I made sure that my handler is attached correctly and waiting, but it's never called. This happens only on Bowser, on FF and Chrome everything happens as expected.

Here are the SDP messages (I suppose it must have something to do with those). On Chrome for example they look different.

Bowser's offer: v=0 o=- 1442807039473499600 1 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic:WMS F+fGCqttcKBXfUsLIdWML4xXyM8gGTGFpG01 m=audio 1 RTP/SAVPF 111 8 0 c=IN IP4 0.0.0.0 a=rtcp-mux a=sendrecv a=rtpmap:111 OPUS/48000/2 a=rtpmap:8 PCMA/8000 a=rtpmap:0 PCMU/8000 a=ssrc:3363789318 cname:user1957962398@host-8460443c a=ssrc:3363789318 msid:F+fGCqttcKBXfUsLIdWML4xXyM8gGTGFpG01 pZvECFn0dXRZRT+BZzQ6ZfwQ9d+2ELO81S+Z a=ice-ufrag:xVdc a=ice-pwd:xc4CvnmGQqWBT2YjD5QgJz a=candidate:19 1 UDP 2013266431 fe80::e864:66ff:fe5c:5f54 50283 typ host a=fingerprint:sha-256 0F:05:9D:19:74:57:4C:A7:4C:1F:D1:A7:5B:95:F5:42:9C:19:D7:EA:C4:CC:72:34:49:A1:3C:94:E6:41:8C:76 a=setup:actpass m=video 61126 RTP/SAVPF 103 100 120 c=IN IP4 192.168.0.128 a=rtcp:50280 IN IP4 192.168.0.128 a=rtcp-mux a=sendrecv a=rtpmap:103 H264/90000 a=rtpmap:100 VP8/90000 a=rtpmap:120 RTX/90000 a=fmtp:103 packetization-mode=1 a=fmtp:120 apt=100;rtx-time=200 a=rtcp-fb:100 nack a=rtcp-fb:103 nack pli a=rtcp-fb:100 nack pli a=rtcp-fb:103 ccm fir a=rtcp-fb:100 ccm fir a=ssrc:2177512475 cname:user1957962398@host-8460443c a=ssrc:2177512475 msid:F+fGCqttcKBXfUsLIdWML4xXyM8gGTGFpG01 jotyiYbInfwDlI31dQuwGs5RcHIuznTt+ggO a=ice-ufrag:i93T a=ice-pwd:1RTqS/xZ0F97gTehEbixct a=candidate:10 1 UDP 2013266431 fe80::e864:66ff:fe5c:5f54 54152 typ host a=candidate:11 1 TCP 1019216383 fe80::e864:66ff:fe5c:5f54 9 typ host tcptype active a=candidate:12 1 TCP 1015022079 fe80::e864:66ff:fe5c:5f54 63778 typ host tcptype passive a=candidate:13 1 UDP 2013266431 192.168.0.128 61126 typ host a=candidate:14 1 TCP 1019216383 192.168.0.128 9 typ host tcptype active a=candidate:15 1 TCP 1015022079 192.168.0.128 63780 typ host tcptype passive a=candidate:16 1 UDP 2013266431 fe80::886:2c8a:72d3:b27e 50281 typ host a=candidate:17 1 TCP 1019216383 fe80::886:2c8a:72d3:b27e 9 typ host tcptype active a=candidate:18 1 TCP 1015022079 fe80::886:2c8a:72d3:b27e 63782 typ host tcptype passive a=candidate:10 2 UDP 2013266430 fe80::e864:66ff:fe5c:5f54 54153 typ host a=candidate:11 2 TCP 1019216382 fe80::e864:66ff:fe5c:5f54 9 typ host tcptype active a=candidate:12 2 TCP 1015022078 fe80::e864:66ff:fe5c:5f54 63779 typ host tcptype passive a=candidate:13 2 UDP 2013266430 192.168.0.128 50280 typ host a=candidate:14 2 TCP 1019216382 192.168.0.128 9 typ host tcptype active a=candidate:15 2 TCP 1015022078 192.168.0.128 63781 typ host tcptype passive a=candidate:16 2 UDP 2013266430 fe80::886:2c8a:72d3:b27e 50282 typ host a=candidate:17 2 TCP 1019216382 fe80::886:2c8a:72d3:b27e 9 typ host tcptype active a=candidate:18 2 TCP 1015022078 fe80::886:2c8a:72d3:b27e 63783 typ host tcptype passive a=fingerprint:sha-256 0F:05:9D:19:74:57:4C:A7:4C:1F:D1:A7:5B:95:F5:42:9C:19:D7:EA:C4:CC:72:34:49:A1:3C:94:E6:41:8C:76 a=setup:actpass m=application 54150 DTLS/SCTP 5000 c=IN IP4 192.168.0.128 a=sendrecv a=ice-ufrag:pNu8 a=ice-pwd:gcQrmt7D5o3g4NVzI08sos a=candidate:1 1 UDP 2013266431 fe80::e864:66ff:fe5c:5f54 54308 typ host a=candidate:2 1 TCP 1019216383 fe80::e864:66ff:fe5c:5f54 9 typ host tcptype active a=candidate:3 1 TCP 1015022079 fe80::e864:66ff:fe5c:5f54 63775 typ host tcptype passive a=candidate:4 1 UDP 2013266431 192.168.0.128 54150 typ host a=candidate:5 1 TCP 1019216383 192.168.0.128 9 typ host tcptype active a=candidate:6 1 TCP 1015022079 192.168.0.128 63776 typ host tcptype passive a=candidate:7 1 UDP 2013266431 fe80::886:2c8a:72d3:b27e 54151 typ host a=candidate:8 1 TCP 1019216383 fe80::886:2c8a:72d3:b27e 9 typ host tcptype active a=candidate:9 1 TCP 1015022079 fe80::886:2c8a:72d3:b27e 63777 typ host tcptype passive a=fingerprint:sha-256 0F:05:9D:19:74:57:4C:A7:4C:1F:D1:A7:5B:95:F5:42:9C:19:D7:EA:C4:CC:72:34:49:A1:3C:94:E6:41:8C:76 a=setup:actpass a=sctpmap:5000 webrtc-datachannel 1024

Server answer:

v=0 o=- 6072310859396819853 2 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic: WMS stream m=audio 9 RTP/SAVPF 111 8 0 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:okovvEM0l16vrnz+ a=ice-pwd:S/krXmc+Z4lb/ymNUNI2zzNg a=fingerprint:sha-256 5C:E6:6B:C6:5A:9A:E3:B1:63:0E:3C:A1:26:CE:CC:5B:92:0F:EF:6D:E1:2A:A9:E6:6E:71:67:E4:2E:B3:AD:CC a=setup:active a=mid:audio a=recvonly a=rtcp-mux a=rtpmap:111 opus/48000/2 a=fmtp:111 minptime=10; useinbandfec=1 a=rtpmap:8 PCMA/8000 a=rtpmap:0 PCMU/8000 a=maxptime:60 m=video 9 RTP/SAVPF 100 120 c=IN IP4 0.0.0.0 a=rtcp:9 IN IP4 0.0.0.0 a=ice-ufrag:wUxPz7k7sbjNCSzA a=ice-pwd:0P84nOudZszipWr24T0CtZNp a=fingerprint:sha-256 5C:E6:6B:C6:5A:9A:E3:B1:63:0E:3C:A1:26:CE:CC:5B:92:0F:EF:6D:E1:2A:A9:E6:6E:71:67:E4:2E:B3:AD:CC a=setup:active a=mid:video a=sendrecv a=rtcp-mux a=rtpmap:100 VP8/90000 a=rtcp-fb:100 ccm fir a=rtcp-fb:100 nack a=rtcp-fb:100 nack pli a=rtpmap:120 rtx/90000 a=fmtp:120 apt=100 a=ssrc-group:FID 887725331 3859995201 a=ssrc:887725331 cname:yf6wM/0S+1bFCmDJ a=ssrc:887725331 msid:stream video a=ssrc:887725331 mslabel:stream a=ssrc:887725331 label:video a=ssrc:3859995201 cname:yf6wM/0S+1bFCmDJ a=ssrc:3859995201 msid:stream video a=ssrc:3859995201 mslabel:stream a=ssrc:3859995201 label:video m=application 9 DTLS/SCTP 5000 c=IN IP4 0.0.0.0 b=AS:30 a=ice-ufrag:EjZEUl43cdDDR12/ a=ice-pwd:HljuqfGkPplx/wynIEQPGrIz a=fingerprint:sha-256 5C:E6:6B:C6:5A:9A:E3:B1:63:0E:3C:A1:26:CE:CC:5B:92:0F:EF:6D:E1:2A:A9:E6:6E:71:67:E4:2E:B3:AD:CC a=setup:active a=mid:data a=sctpmap:5000 webrtc-datachannel 1024

[stefhak]

What does the candidates supplied by the server look like? And can your server digest the bowser candidates when they are part of the SDP offer (and not "trckled")?

[pantaoran]

Thanks for your quick reaction! My server logged these candidates. Server is based on libjingle by the way. Edit: censored some real IPs. Edit2: the remaining non-local IPs 199.etc are the end point of the commercial VPN that I use. I'm in China so using internet without VPN is not really an option.

candidate:2991111282 1 udp 2122260223 192.168.0.53 37448 typ host generation 0 candidate:2441410931 1 udp 2122194687 [censored].42.1 43774 typ host generation 0 candidate:2991111282 1 udp 2122260223 192.168.0.53 46734 typ host generation 0 candidate:2441410931 1 udp 2122194687 [censored].42.1 47710 typ host generation 0 candidate:2991111282 1 udp 2122260223 192.168.0.53 57031 typ host generation 0 candidate:2441410931 1 udp 2122194687 [censored].42.1 36789 typ host generation 0 candidate:4241178754 1 tcp 1518280447 192.168.0.53 34424 typ host tcptype passive generation 0 candidate:3741779331 1 tcp 1518214911 [censored].42.1 54986 typ host tcptype passive generation 0 candidate:4241178754 1 tcp 1518280447 192.168.0.53 33026 typ host tcptype passive generation 0 candidate:3741779331 1 tcp 1518214911 [censored].42.1 45416 typ host tcptype passive generation 0 candidate:4241178754 1 tcp 1518280447 192.168.0.53 42945 typ host tcptype passive generation 0 candidate:3741779331 1 tcp 1518214911 [censored].42.1 40435 typ host tcptype passive generation 0 candidate:1146668454 1 udp 1686052607 199.48.230.176 37448 typ srflx raddr 192.168.0.53 rport 37448 generation 0 candidate:953906854 1 udp 1685987071 199.48.230.176 43774 typ srflx raddr [censored].42.1 rport 43774 generation 0

[Firegarden]

What does the calling code look like? Are you using adapter? For example are you making sure you have pc.signalingState === STABLE before you create offer?

[stefhak]

Just to make sure: there are no errors/warnings fired?

On Mon, Sep 21, 2015 at 11:08 AM, Robert Sherrill notifications@github.com wrote:

What does the calling code look like? Are you using adapter?

— Reply to this email directly or view it on GitHub https://github.com/EricssonResearch/bowser/issues/53#issuecomment-141919008 .

[stefhak]

The candidates sent from the server: when you send them outside the SDP they should contain m-line indeces, right? So bowser can know if a certain candidate pertains to audio, video or data.

I also note that audio is reconly in the answer (while video is sendrecv). I don't know what is going on, I'm just tossing ideas.

On Mon, Sep 21, 2015 at 11:38 AM, stefan hakansson stefhak@gmail.com wrote:

Just to make sure: there are no errors/warnings fired?

On Mon, Sep 21, 2015 at 11:08 AM, Robert Sherrill < notifications@github.com> wrote:

What does the calling code look like? Are you using adapter?

— Reply to this email directly or view it on GitHub https://github.com/EricssonResearch/bowser/issues/53#issuecomment-141919008 .

[pantaoran]

@Firegarden No, I'm not using adapter. That thing looks like a mess and has no Bowser support anyway. And yes, the signalingState is stable before I start, I just doublechecked.

@stefhak There are no errors, just a couple of deprecation warnings like "getUserMedia() is deprecated on insecure origins" and such. Server is programmed to not send any audio back, we are just augmenting the video only. Not sure what you mean about the m-line indices, need to think about that.

I'm currently trying to get any infos out of the getStats() interface to see if it can confirm that my pc is actually connected and how.

[pantaoran]

Ehm, does Bowser actually support getStats? When I try to call it it says the function is undefined...

[stefanalund]

@pantaoran no, sorry. Please create a new Issue so that people can track it. Here is a good overview of what is supported in Bowser (and other browsers): http://iswebrtcreadyyet.com/

[Firegarden]

Bowser seems to load then crash as noted here on any of my ios devices. Is there a specific ios version that is required? I have ipad mini 1 v7.1.1, ipad2 v9 iphone6 v8

I think bowser is amazing I just don't understand how to get a stable setup so I can demo my browser based webrtc app using an ios device. Please advise?

[stefanalund]

Bowser currently does not work on 32-bit devices such as your iPad mini and iPad 2. The iPhone 6 should work because it is 64-bit @Firegarden.

[Firegarden]

Jolly good sir. Good show. For he's a jolly good fellow, for he's a jolly good fellow For he's a jolly good fellow (pause), and so say all of us And so say all of us, and so say all of us For he's a jolly good fellow, for he's a jolly good fellow For he's a jolly good fellow (pause), and so say all of us!

[pantaoran]

@stefanalund thanks for the overview table. but according to that, bowser doesn't support dataChannels, however I have used dataChannels with Bowser in other demos successfully. how up to date is that table?

[stefanalund]

Huh, that's not good :) It's quite simple to fix though, just edit the table and send a PR to their repo here: https://github.com/webrtcftw/iswebrtcreadyyet.com @fippo usually lands them quickly.

[pantaoran]

so to get back to my original problem: I found the m-line indices that @stefhak mentioned, and now I even understand what they are :-) they seem correct to me. for audio candidates they are 0, for video candidates 1, and for data candidates they are 2.

any other ideas why I might never get an onaddstream event? is there any corresponding error event I should listen for?

my data channel works fine btw, I get onopen can can transmit messages there...

[pantaoran]

I'd also like to note that I get n'either the oniceconnectionstatechange event nor the onsignalingstatechange event on the connection with Bowser. my listeners for those events work fine on chrome and FF.

Edit: no wonder I dont get oniceconnectionstatechange, the state never actually changes. by logging the state at other moments I found that it is always NEW and never becomes CHECKING or CONNECTED. finally I have a lead again to debug this further. Edit2: the same also applies for the icegatheringstate, it is also always NEW and never changes to anything else :-(

[pantaoran]

and now I've come full circle. in my first issue here (#51) I said that the SDP from Bowser could not be parsed on the server side due to multiple m=audio and m=video lines, so the workaround was to leave this part

this.configOffer={ "offerToReceiveVideo": 1, "offerToReceiveAudio": 1 }

out of the code when I detect Bowser as the user agent. But now this bites me in the ass, because according to the internet (for example here http://stackoverflow.com/q/27489881/1446479) this will cause the connection to never change icegatheringstate or iceconnectionstate. Which makes me wonder how this can work with other sites? Hmmm.....

[stefanalund]

@pantaoran our demo app is available here for reference: https://github.com/EricssonResearch/openwebrtc-examples/tree/master/web

[stefhak]

@pantaoran I'll look into this tomorrow and hopefully give some answer that is useful. I'm not sure we have implemented the state change events (will have to look at the source).

I don't think you should care much about the stackoverflow piece, that seems outdated. Looking at the SDPs at the top of this issue, they look OK to me. And the fact that the data channel works makes me believe that ICE and DTLS works fine.

I'm a bit unsure about the 'a=rtcp:9' lines in the SDP from the server, since rtcp-mux is offered by Bowser and accepted by the server, I don't think they (i.e those lines) should be there in the answer.

[pantaoran]

that would be great, thanks!

[pantaoran]

I'm now comparing and analyzing your demo app in detail, and I have a question. When looking at this file https://github.com/EricssonResearch/openwebrtc-examples/blob/master/web/client/main.js it seems that you're not using the SDP at all, instead you rely on an external sdp.js file which has a parse() method and generates something you call "sessionDescription" in the code (line 272 in the linked file), and then you use that to send over the signaling channel instead of the sdp. Can you explain to me why this is and what exactly that does? What kind of format is this and would my libjingle server also understand this?

Edit: I understand now that I can just leave that sdp.js file out and then it will fall back to using the original sdp. But that doesn't change the fact that that demo doesn't actually work (see next comment).

[pantaoran]

I've been playing around with your demo after finally deploying it locally and not just your hosted version. I can confirm now that with your demo also, the state changes don't seem to happen, they remain in state NEW.

Furthermore I have to say: I could not get that demo to work properly for video between Bowser and my Android Chrome. Depending on who initiates the call I can sometimes get the video on one side, but so far I've never succeeded in making a bidirectional video call where both can see each other. Can someone try to confirm this?

[stefhak]

I've looked into the code, and it seems owr does not update those states. Could you file an issue?

[pantaoran]

I will. Could you confirm that the demo app here https://github.com/EricssonResearch/openwebrtc-examples/tree/master/web is unable to produce a bidirectional video connection (when one of the peers is Bowser)?

[stefhak]

Thanks.

Actually, I use that app quite a lot and it works for me (but I use it with Safari on Mac OS X, not bowser) when the other client is Chrome or FF (the (not so) amusing part is that safari+owr <-> safari+owr on the same machine fails, only one way video for those cases).

Update: just tested Bowser on my iPhone5s (iOS9) <-> FF (41), and http://demo.openwebrtc.io:38080 works with video both ways (regardless of whether "call" is pressed on Bowser or Firefox)

[pantaoran]

Thanks for the info. When you say owr, is that a synonym for Bowser? If not, what exactly does owr stand for?

Good to know about the two-way video. So far I've been testing on iPad Air 2 (iOS 8) and Chrome 45. I'll see that I can get my hands on an iOS 9 device for some testing tomorrow.

[stefhak]

owr means OpenWebRTC, the code Bowser is based on.

Regarding test, don't hold your breath. It worked fine with iOS 8 as well for me. Have you tested with Firefox?

[pantaoran]

I have made an interesting realization: I had always selected all three boxes (audio/video/chat) and the audio/video never worked bidirectionally (Bowser never fired the onaddstream event). However, when I DON'T select the chat box then the bidirectional video works (for me again with Bowser iOS 8 and Chrome 45 Android)!

The DataChannel seems to somehow interfere with the other stuff, preventing that from working correctly. Could you let me know which boxes you had ticked when you tested it yesterday?

Thanks for all your help!

[stefhak]

I can confirm. Bowser <-> FF (41, desktop, mac os x) gives one-way video if the chat box is ticked (and the result is the same regardless if the "call" button is clicked on Bowser or FF). The remote video pops up in FF, but not in Bowser. If "chat" is not selected things seem to work.

Update: I get the same result when the non-FF side is openwebrtc-daemon. In other words, this is not Bowser specific but seems to be an issue with owr. @stefanalund @superdump - something to investigate?

[y-element]

Just wanted to inform that, I also observe this issue with bowser 0.6.1 (would like to inform any fixes but currently I'm just disabling datachannels on my application).