search

versatica / mediasoup-client

mediasoup client side JavaScript library
https://mediasoup.org
ISC License
583 stars 236 forks source link

remote track not found in react native #93

Closed zhangzhx666 closed 4 years ago

zhangzhx666 commented 4 years ago

In react native when I call transport.consume(), An error occurred with 'remote track not found'.

ibc commented 4 years ago

Hard to help with that tiny issue description. Please get the full logs with the "remote" SDP that mediasoup-client is generating and so on.

zhangzhx666 commented 4 years ago

@ibc when I pass params 'kind=audio' to transport.consume(),it'ok.when i pass params 'kind=video',it's failed. info: 'remote track not found'. offer sdp: v=0 o=mediasoup-client 10000 2 IN IP4 0.0.0.0 s=- t=0 0 a=ice-lite a=fingerprint:sha-512 FB:E7:16:80:77:A3:95:87:F0:1A:47:90:6E:2E:10:EF:B6:16:15:02:BD:D0:65:90:51:22:5F:64:35:E3:38:6F:E5:80:92:CA:C8:0F:4F:7A:1B:02:70:9E:C1:AC:B7:F9:54:78:4C:87:7F:BE:16:06:0C:2F:7A:3C:0B:89:28:FA a=msid-semantic: WMS * a=group:BUNDLE audio video m=audio 7 UDP/TLS/RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1;sprop-stereo=1;usedtx=1 a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:audio a=sendonly a=ice-ufrag:ouw3wdedlpgmt0mn a=ice-pwd:lbx096r662x3uhb8agyxuj47zru872w1 a=candidate:udpcandidate 1 udp 1076302079 39.107.252.171 45248 typ host a=end-of-candidates a=ice-options:renomination a=ssrc:171283064 cname:P/ERT6t4K9mbkzN6 a=ssrc:171283064 msid:P/ERT6t4K9mbkzN6 e216bba6-f574-4055-bc77-ee2a03d4a934 a=rtcp-mux a=rtcp-rsize m=video 7 UDP/TLS/RTP/SAVPF 101 102 c=IN IP4 127.0.0.1 a=rtpmap:101 VP8/90000 a=rtpmap:102 rtx/90000 a=fmtp:102 apt=101 a=rtcp-fb:101 goog-remb a=rtcp-fb:101 ccm fir a=rtcp-fb:101 nack a=rtcp-fb:101 nack pli a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:6 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:11 urn:3gpp:video-orientation a=extmap:12 urn:ietf:params:rtp-hdrext:toffset a=setup:actpass a=mid:video a=sendonly a=ice-ufrag:ouw3wdedlpgmt0mn a=ice-pwd:lbx096r662x3uhb8agyxuj47zru872w1 a=candidate:udpcandidate 1 udp 1076302079 39.107.252.171 45248 typ host a=end-of-candidates a=ice-options:renomination a=ssrc:713358383 cname:P/ERT6t4K9mbkzN6 a=ssrc:713358383 msid:P/ERT6t4K9mbkzN6 0a59d844-53ad-4871-94a9-1770db2a84f4 a=ssrc:753697393 cname:P/ERT6t4K9mbkzN6 a=ssrc:753697393 msid:P/ERT6t4K9mbkzN6 0a59d844-53ad-4871-94a9-1770db2a84f4 a=ssrc-group:FID 713358383 753697393 a=rtcp-mux a=rtcp-rsize

anwser sdp: v=0 o=- 8969421427584378601 3 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic: WMS a=group:BUNDLE audio video m=audio 39873 UDP/TLS/RTP/SAVPF 100 c=IN IP4 192.168.31.112 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;stereo=1;useinbandfec=1 a=rtcp:9 IN IP4 0.0.0.0 a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:active a=mid:audio a=recvonly a=ice-ufrag:swGZ a=ice-pwd:Ze/DWk7EcmH1tgv0NNN36Uds a=fingerprint:sha-256 62:AB:39:6D:B0:14:46:F5:65:DC:88:9C:28:D3:4D:1A:FA:FB:51:FF:60:D1:8B:CC:46:5F:10:85:40:8E:B3:48 a=candidate:1045070234 1 udp 2122260223 192.168.31.112 39873 typ host generation 0 network-id 3 network-cost 10 a=ice-options:trickle a=rtcp-mux m=video 9 UDP/TLS/RTP/SAVPF 101 102 c=IN IP4 0.0.0.0 a=rtpmap:101 VP8/90000 a=rtpmap:102 rtx/90000 a=fmtp:102 apt=101 a=rtcp:9 IN IP4 0.0.0.0 a=rtcp-fb:101 goog-remb a=rtcp-fb:101 ccm fir a=rtcp-fb:101 nack a=rtcp-fb:101 nack pli a=extmap:12 urn:ietf:params:rtp-hdrext:toffset a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:11 urn:3gpp:video-orientation a=extmap:6 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=setup:active a=mid:video a=recvonly a=ice-ufrag:swGZ a=ice-pwd:Ze/DWk7EcmH1tgv0NNN36Uds a=fingerprint:sha-256 62:AB:39:6D:B0:14:46:F5:65:DC:88:9C:28:D3:4D:1A:FA:FB:51:FF:60:D1:8B:CC:46:5F:10:85:40:8E:B3:48 a=ice-options:trickle a=rtcp-mux a=rtcp-rsize

ibc commented 4 years ago

I do not see the logs so I don't know which parameters are being passed to transport.consume(), so hard to figure out what's happening.

zhangzhx666 commented 4 years ago

which parameters?rtpParameters? my call is: this._recvTransport.consume( { id, producerId, kind, rtpParameters, codecOptions, appData : { ...appData, peerId } // Trick. });

ibc commented 4 years ago

That's not your call but the API definition. I mean real values and the corresponding SDPs after using them.

zhangzhx666 commented 4 years ago

@ibc Thanks for your reply. The order of my call:

  1. this._recvTransport.consume( "54feb924-7425-4a68-86f3-575ea72d4330", "c7ae4934-6f03-4d79-ab75-04584f05aa80", "audio", rtpParameters, {opusStereo : 1}, appData : {peerId: "nvggarv6"} ). rtpParameters={codecs: Array(1), headerExtensions: Array(2), encodings: Array(1), rtcp: {cname: "KwOJp2tzXPIsmzsW", reducedSize: true, mux: true}} //detailed rtpParameters rtpParameters.codecs[1]=[ { mimeType: "audio/opus", clockRate: 48000, payloadType: 100, channels: 2, parameters:{minptime: 10, useinbandfec: 1, sprop-stereo: 1, usedtx: 1}, rtcpFeedback:[] }]; rtpParameters.encodings[1]=[{ssrc: 325330938}]; rtpParameters.headerExtensions[4]=[ {uri: "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time", id: 4}, {uri: "urn:ietf:params:rtp-hdrext:ssrc-audio-level", id: 10} ]; 2.this._recvTransport.consume( "6b51120e-61bb-45cd-8dbb-853535b0c35c", "4904ad67-b257-467a-b53d-b62e883546cf", "video", rtpParameters, null, appData : {peerId: "nvggarv6" } ). rtpParameters={codecs: Array(1), headerExtensions: Array(2), encodings: Array(1), rtcp: {cname: "KwOJp2tzXPIsmzsW", reducedSize: true, mux: true}} //detailed rtpParameters rtpParameters.codecs[2]=[ { clockRate:90000, mimeType:"video/VP8", payloadType:101, parameters:{}, rtcpFeedback:[{type: "goog-remb"},{type: "ccm", parameter: "fir"},{type: "nack"},{type: "nack", parameter: "pli"}] }, { clockRate:90000, mimeType:"video/rtx", payloadType:102, parameters:{apt: 101}, rtcpFeedback:[] } ]; rtpParameters.encodings[1]=[{{ssrc: 348596064, rtx: {ssrc: 244868409}, scalabilityMode: "S3T3"}}]; rtpParameters.headerExtensions[4]=[ {uri: "http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time", id: 4}, {uri: "http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07", id: 6}, {uri: "urn:3gpp:video-orientation", id: 11}, {uri: "urn:ietf:params:rtp-hdrext:toffset", id: 12} ];
zhangzhx666 commented 4 years ago

The first step sdp: offer sdp: v=0 o=mediasoup-client 10000 1 IN IP4 0.0.0.0 s=- t=0 0 a=ice-lite a=fingerprint:sha-512 B8:1A:1B:9E:26:AC:46:66:AF:5C:A2:39:BD:F2:FA:E3:43:65:0B:B4:58:13:96:BC:A7:A5:C2:C1:FB:76:8E:B5:0D:15:86:D6:C1:BC:88:80:75:C7:71:A0:82:FE:69:43:B7:6A:98:79:86:2C:49:87:E5:B8:F4:CE:78:53:95:9B a=msid-semantic: WMS * a=group:BUNDLE audio m=audio 7 UDP/TLS/RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1;sprop-stereo=1;usedtx=1 a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:audio a=sendonly a=ice-ufrag:g2kn4hid1pgax4u3 a=ice-pwd:tbpc9efqqmt3etnka0ydmz4rfja03es5 a=candidate:udpcandidate 1 udp 1076302079 39.107.252.171 45704 typ host a=end-of-candidates a=ice-options:renomination a=ssrc:325330938 cname:KwOJp2tzXPIsmzsW a=ssrc:325330938 msid:KwOJp2tzXPIsmzsW 54feb924-7425-4a68-86f3-575ea72d4330 a=rtcp-mux a=rtcp-rsize answer sdp: v=0 o=- 5490358975940665396 2 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic: WMS a=group:BUNDLE audio m=audio 9 UDP/TLS/RTP/SAVPF 100 c=IN IP4 0.0.0.0 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1;stereo=1 a=rtcp:9 IN IP4 0.0.0.0 a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:active a=mid:audio a=recvonly a=ice-ufrag:Gtgm a=ice-pwd:Xe+ZrunYuy6mGmWdav6KfOAS a=fingerprint:sha-256 6C:DB:37:2E:92:E7:9B:28:77:A9:86:EF:3A:67:86:C2:EF:F7:90:5F:CA:C6:F8:B3:B0:44:3E:64:78:93:A0:C3 a=ice-options:trickle a=rtcp-mux

zhangzhx666 commented 4 years ago

The second step sdp: offer sdp: v=0 o=mediasoup-client 10000 2 IN IP4 0.0.0.0 s=- t=0 0 a=ice-lite a=fingerprint:sha-512 B8:1A:1B:9E:26:AC:46:66:AF:5C:A2:39:BD:F2:FA:E3:43:65:0B:B4:58:13:96:BC:A7:A5:C2:C1:FB:76:8E:B5:0D:15:86:D6:C1:BC:88:80:75:C7:71:A0:82:FE:69:43:B7:6A:98:79:86:2C:49:87:E5:B8:F4:CE:78:53:95:9B a=msid-semantic: WMS * a=group:BUNDLE audio video m=audio 7 UDP/TLS/RTP/SAVPF 100 c=IN IP4 127.0.0.1 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;useinbandfec=1;sprop-stereo=1;usedtx=1 a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:actpass a=mid:audio a=sendonly a=ice-ufrag:g2kn4hid1pgax4u3 a=ice-pwd:tbpc9efqqmt3etnka0ydmz4rfja03es5 a=candidate:udpcandidate 1 udp 1076302079 39.107.252.171 45704 typ host a=end-of-candidates a=ice-options:renomination a=ssrc:325330938 cname:KwOJp2tzXPIsmzsW a=ssrc:325330938 msid:KwOJp2tzXPIsmzsW 54feb924-7425-4a68-86f3-575ea72d4330 a=rtcp-mux a=rtcp-rsize m=video 7 UDP/TLS/RTP/SAVPF 101 102 c=IN IP4 127.0.0.1 a=rtpmap:101 VP8/90000 a=rtpmap:102 rtx/90000 a=fmtp:102 apt=101 a=rtcp-fb:101 goog-remb a=rtcp-fb:101 ccm fir a=rtcp-fb:101 nack a=rtcp-fb:101 nack pli a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:6 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=extmap:11 urn:3gpp:video-orientation a=extmap:12 urn:ietf:params:rtp-hdrext:toffset a=setup:actpass a=mid:video a=sendonly a=ice-ufrag:g2kn4hid1pgax4u3 a=ice-pwd:tbpc9efqqmt3etnka0ydmz4rfja03es5 a=candidate:udpcandidate 1 udp 1076302079 39.107.252.171 45704 typ host a=end-of-candidates a=ice-options:renomination a=ssrc:348596064 cname:KwOJp2tzXPIsmzsW a=ssrc:348596064 msid:KwOJp2tzXPIsmzsW 6b51120e-61bb-45cd-8dbb-853535b0c35c a=ssrc:244868409 cname:KwOJp2tzXPIsmzsW a=ssrc:244868409 msid:KwOJp2tzXPIsmzsW 6b51120e-61bb-45cd-8dbb-853535b0c35c a=ssrc-group:FID 348596064 244868409 a=rtcp-mux a=rtcp-rsize answer sdp: v=0 o=- 5490358975940665396 3 IN IP4 127.0.0.1 s=- t=0 0 a=msid-semantic: WMS a=group:BUNDLE audio video m=audio 42894 UDP/TLS/RTP/SAVPF 100 c=IN IP4 192.168.31.112 a=rtpmap:100 opus/48000/2 a=fmtp:100 minptime=10;stereo=1;useinbandfec=1 a=rtcp:9 IN IP4 0.0.0.0 a=extmap:10 urn:ietf:params:rtp-hdrext:ssrc-audio-level a=setup:active a=mid:audio a=recvonly a=ice-ufrag:Gtgm a=ice-pwd:Xe+ZrunYuy6mGmWdav6KfOAS a=fingerprint:sha-256 6C:DB:37:2E:92:E7:9B:28:77:A9:86:EF:3A:67:86:C2:EF:F7:90:5F:CA:C6:F8:B3:B0:44:3E:64:78:93:A0:C3 a=candidate:1045070234 1 udp 2122260223 192.168.31.112 42894 typ host generation 0 network-id 3 network-cost 10 a=ice-options:trickle a=rtcp-mux m=video 9 UDP/TLS/RTP/SAVPF 101 102 c=IN IP4 0.0.0.0 a=rtpmap:101 VP8/90000 a=rtpmap:102 rtx/90000 a=fmtp:102 apt=101 a=rtcp:9 IN IP4 0.0.0.0 a=rtcp-fb:101 goog-remb a=rtcp-fb:101 ccm fir a=rtcp-fb:101 nack a=rtcp-fb:101 nack pli a=extmap:12 urn:ietf:params:rtp-hdrext:toffset a=extmap:4 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time a=extmap:11 urn:3gpp:video-orientation a=extmap:6 http://tools.ietf.org/html/draft-ietf-avtext-framemarking-07 a=setup:active a=mid:video a=recvonly a=ice-ufrag:Gtgm a=ice-pwd:Xe+ZrunYuy6mGmWdav6KfOAS a=fingerprint:sha-256 6C:DB:37:2E:92:E7:9B:28:77:A9:86:EF:3A:67:86:C2:EF:F7:90:5F:CA:C6:F8:B3:B0:44:3E:64:78:93:A0:C3 a=ice-options:trickle a=rtcp-mux a=rtcp-rsize

zhangzhx666 commented 4 years ago

Receiving the audio is ok, receiving the video failed.Tips:'remote track not found'.

ibc commented 4 years ago

Thanks, will check next week if I get some time.

zhangzhx666 commented 4 years ago

Thanks,waiting for your check results.Happy weekend！

ibc commented 4 years ago

I'm afraid I'm super busy and cannot give so much support as people is lately requesting. I don't have a react-native app to test this. I'd appreciate if you could dig into the code and check what is wrong.

ibc commented 4 years ago

@saghul may you please confirm whether this should work or not? It seems that it works for the first remote audio track BTW.

Here after the second SDP O/A a new m=video has been added in the remote SDP offer (simplified):

m=video 7 UDP/TLS/RTP/SAVPF 101 102
a=mid:video
a=sendonly
a=ssrc:348596064 cname:KwOJp2tzXPIsmzsW
a=ssrc:348596064 msid:KwOJp2tzXPIsmzsW 6b51120e-61bb-45cd-8dbb-853535b0c35c
a=ssrc:244868409 cname:KwOJp2tzXPIsmzsW
a=ssrc:244868409 msid:KwOJp2tzXPIsmzsW 6b51120e-61bb-45cd-8dbb-853535b0c35c
a=ssrc-group:FID 348596064 244868409

Here the code to retrieve the generated new remote video track:

https://github.com/versatica/mediasoup-client/blob/v3/lib/handlers/ReactNative.js#L513

const answerDesc = new RTCSessionDescription(answer);

await this._pc.setLocalDescription(answerDesc);

const stream = this._pc.getRemoteStreams()
    .find((s) => s.id === streamId);
const track = stream.getTrackById(localId);

if (!track)
    throw new Error('remote track not found');

The resulting error is that stream does exist but track does not.

Take into account that:

saghul commented 4 years ago

I think it should... are any tracks detected there? What ids do they have?

ibc commented 4 years ago

I don't have any react-native app to test. @zhangzhx666 can you please modify the code of handlers/ReactNative.js and print the id of all the entries returned by this._pc.getRemoteStreams() and also the id of each track in stream.getTrack() (for all the streams)?

zhangzhx666 commented 4 years ago

@ibc @saghul The reason of error is that audio track and video track have the same stream id in android . There is the similar isssue that you submitted. react-native-webrtc/react-native-webrtc#401

ibc commented 4 years ago

Good catch. Yes, that's the problem. It seems that such a bug (which is a problem in ObjC bindings of libwebrtc) will be there forever and ever.

@zhangzhx666 @saghul do you know how to detect (in JS) if the react-native app is running in iOS?

ibc commented 4 years ago

Ok @zhangzhx666 can you please check the new "issue93" branch of mediasoup-client?

ibc commented 4 years ago

BTW this is the approach: https://github.com/versatica/mediasoup-client/commit/30596d2f67fb334e86a4302c02c91086509b30de?ts=2

Basically, if React-Native has been detected (so we are in handles/ReactNative.js) and iOS has been detected, then it creates a random streamId for every remote track, no matter it belongs to the same sender as any other remote track. This is, audio and video Consumers from the same Producer will have different streamId so the remote SDP will have different a=msid and will avoid the painful bug of ObjC in libwebrtc.

@saghul does it look fine for you?

zhangzhx666 commented 4 years ago

@ibc OK.Thanks for your helps.

ibc commented 4 years ago

Does it work?

ibc commented 4 years ago

Umm, the approach in that "issue93" is not good since it breaks mediasoup-client in the browser when using browserify or similar:

Error: Cannot find module 'react-native'

@saghul is there any way to check if this is iOS in React-Native without using the react-native module?

ibc commented 4 years ago

Please @zhangzhx666, clarify if this happens with Android and/or iOS, and whether it also happens with the new branch "issue93".

In the other side, there is a PR for react-native-webrtc that may make this work, although I'll comment on it because it's not 100% clear to me:

https://github.com/react-native-webrtc/react-native-webrtc/pull/564

zhangzhx666 commented 4 years ago

@ibc I only have an android app. The new branch "issue93" cannot work in android.I modified the "streamId" according to the approach in that "issue93",it works ok now. async receive({ id, kind, rtpParameters }) { logger.debug('receive() [id:%s, kind:%s]', id, kind); const localId = id; const mid = kind; // NOTE: In Mobile we cannot reuse the same remote MediaStream for new remote // tracks. So force the streamId to be different. const streamId = recv-stream-${id}; //rtpParameters.rtcp.cname; this._remoteSdp.receive( { mid, kind, offerRtpParameters : rtpParameters, streamId, trackId : localId });

ibc commented 4 years ago

The new branch "issue93" cannot work in android

What does it mean that "it cannot work in Android"?

BTW I already modified the branch to also affect Android: https://github.com/versatica/mediasoup-client/commit/155cb5ca835eabc16dade71e50534c868031d8e2

Now, the thing is that we may not need this change in mediasoup-client since it's not good (libwebrtc uses a=msid to sync audio and video remote tracks from the same remote source, so these changes in mediasoup-client avoid it in React-Native and may produce lipsync issues). So probably the solution is in react-native-webrtc side:

@zhangzhx666 can you use mediasoup-client "v3" branch and this fork of react-native-webrtc?: react-native-webrtc/react-native-webrtc#564

zhangzhx666 commented 4 years ago

@ibc I will test next week with the mediasoup-client "v3" branch and this fork of react-native-webrtc:react-native-webrtc/react-native-webrtc#564.

ibc commented 4 years ago

It won't work in Android. @saghul is working on it. Please subscribe to https://github.com/react-native-webrtc/react-native-webrtc/pull/564 and wait for news.

In the meanwhile I've deleted the "issue93" branch and added the hack into mediasoup-client 3.2.9, for both Android and iOS, so it should work now. Once react-native-webrtc fixes its issue I'll update this and remove the hack.

ashishvista commented 4 years ago

Receiving the audio is ok, receiving the video failed.Tips:'remote track not found'.

Receiving the audio is ok, receiving the video failed.Tips:'remote track not found'.

facing same issue. @ibc please help.

ibc commented 4 years ago

There are LOT of comments above explaining the issue, how is it fixed and how the way to go is in the future. Please don't just report "the same" without confirming you are aware of comments above.

ashishvista commented 4 years ago

There are LOT of comments above explaining the issue, how is it fixed and how the way to go is in the future. Please don't just report "the same" without confirming you are aware of comments above.

thanks @ibc its working with the above suggested comments. I am now able to get remote video and audiotracks but now these tracks cannot be played in react native rtcview tag. Local stream are being played but remote stream showing blank without any errors. Issue opened in react native webrtc repo. https://github.com/react-native-webrtc/react-native-webrtc/issues/717

ibc commented 4 years ago

@ashishvista, thanks for your contribution to react-antive-webrtc. So I assume this works now and this issue can be closed. Please notify otherwise.