search

centricular / gstwebrtc-demos

Superseded by https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-examples/webrtc - Demo apps for using gstreamer's webrtc implementation
https://gitlab.freedesktop.org/gstreamer/gstreamer/-/tree/main/subprojects/gst-examples/webrtc
BSD 2-Clause "Simplified" License
490 stars 197 forks source link

How to use webrtc-unidirectional-h264? #172

Closed zingfrid closed 3 years ago

zingfrid commented 4 years ago

I've compiled it with GST 1.17 (master) and start it. It shows:

WebRTC page link: http://127.0.0.1:57778/

When I go to link in browser, I have this message in the terminal: Processing new websocket connection 0x556eb535f920

And that's all. Nothing happens. Browser' javascript console shows nothing too.

What is the correct way to run it? Should I start the signaling server before?

thaytan commented 4 years ago

The unidirectional example is a standalone simple HTTP server, it doesn't require the signalling service. It's also a really minimal example and not handling any pipeline errors properly - so my first guess is that you are missing a plugin and it can't build the sender pipeline.

Run the example with GST_DEBUG=3 like: GST_DEBUG=3 ./webrtc-unidirectional-h264

and see if any warning or error level messages are printed.

zingfrid commented 4 years ago

Thank you very much for your help. I've managed to make it works with videotestsrc and firefox. But now, when I'm trying to set rtspsrc instead of videotestsrc it stops working. I don't know why, but on the almost same offer firefox sends completely different answer: Here is videotestsrc offer and firefox answer:


 Negotiation offer created:
v=0
o=- 605351187735251100 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:PlDlYGWc3Cv4SLQXpXNruB+CPswDgqHT
a=ice-pwd:epaey6zLfRWKcydN+OwmSGyqeHqk5KTt
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=42c00d;sprop-parameter-sets=Z0LADdoFB+wFqDAINSgAAAMACAAAAwHkeKFV,aM48gA==
a=ssrc:3594085650 msid:user1213536650@host-936ce970 webrtctransceiver0
a=ssrc:3594085650 cname:user1213536650@host-936ce970
a=mid:video0
a=fingerprint:sha-256 8E:E8:7F:C0:E9:98:44:78:F2:EF:D1:A4:2A:41:4B:58:94:67:76:63:04:72:8F:BA:CE:AE:75:92:CA:ED:33:36

Received SDP:
v=0
o=mozilla...THIS_IS_SDPARTA-75.0 4305277589614964782 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 4B:6F:E2:E9:A0:EE:EE:A8:29:93:AA:5C:67:44:5C:68:17:3A:59:BF:4C:46:D3:F8:B5:BE:A5:58:39:D6:50:64
a=ice-options:trickle
a=msid-semantic:WMS *
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=recvonly
a=fmtp:96 profile-level-id=42e00d;level-asymmetry-allowed=1;packetization-mode=1
a=ice-pwd:8651e0cf0720238624c0674dd56b97d5
a=ice-ufrag:6d3d35bb
a=mid:video0
a=rtcp-fb:96 nack pli
a=rtcp-mux
a=rtpmap:96 H264/90000
a=setup:active

Here is with rtspsrc:

Negotiation offer created:
v=0
o=- 5886926381932552102 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:8j2pXxYTwEEL9cwzDTprzQOuGx03gk2S
a=ice-pwd:azJw/arzC58TbxI83ABTvvZNgu9m/aSh
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:20
a=fmtp:96 packetization-mode=1;profile-level-id=4d4028;sprop-parameter-sets=Z01AKJWgHgCJ+WbIAAADAAgAAAMBRCA=,aO48gA==
a=ssrc:2919030461 msid:user3184521545@host-62db5e00 webrtctransceiver0
a=ssrc:2919030461 cname:user3184521545@host-62db5e00
a=mid:video0
a=fingerprint:sha-256 AF:CE:87:C7:A5:27:9E:6E:21:3F:4B:8D:34:F7:2A:78:FC:4D:07:39:EA:64:71:19:61:D5:40:43:22:E9:AC:6E

Received SDP:
v=0
o=mozilla...THIS_IS_SDPARTA-75.0 4482974859843286918 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 57:3A:E7:73:22:8C:ED:E7:D5:A5:EB:2F:71:80:43:AF:14:DB:69:3A:B1:55:2B:12:CE:7E:F8:28:0D:E0:03:92
a=ice-options:trickle
a=msid-semantic:WMS *
m=video 0 UDP/TLS/RTP/SAVPF 120
c=IN IP4 0.0.0.0
a=inactive
a=mid:video0
a=rtpmap:120 VP8/90000

I know, that it is out of this issue, but I was not able to google anything similar. Any thought?

thaytan commented 4 years ago

Firefox is fussy about the profile-level-id in the offer, which is determined by the H.264 profile and level. The 'video/x-h264,profile=constrained-baseline' caps filter constrains that partly and works for videotestsrc with the 640x360 @ 15fps video.

Changing it to 'video/x-h264,profile=constrained-baseline,level=(string)4' works for me here with 1080p 30fps video and Firefox

zingfrid commented 4 years ago

I'm trying to stream h264 directly from IP camera. So my pipeline, ideally, should look like:

receiver_entry->pipeline = gst_parse_launch ("webrtcbin name=webrtcbin stun-server=stun://" STUN_SERVER " "
       " ! rtspsrc location=rtsp://admin:admin@192.168.1.12 buffer-mode=0 latency=10"
       " ! webrtcbin. ", &error);

No h264depay - h264pay required, as I already have rtp packed h264 stream. So, I'm not very clear where I should set 'video/x-h264,profile=constrained-baseline,level=(string)4' caps.

zingfrid commented 4 years ago

I've found out. That in such case I should add webrtc transiver manually:

GstWebRTCRTPTransceiver *trans = NULL;
GstCaps *caps = 
gst_caps_new_simple (
 "application/x-rtp",
"media", G_TYPE_STRING, "video",
"encoding-name", G_TYPE_STRING, "H264",
"clock-rate", G_TYPE_INT, 90000,
"payload", G_TYPE_INT, RTP_PAYLOAD_TYPE,
NULL);
g_signal_emit_by_name( receiver_entry->webrtcbin, "add-transceiver", GST_WEBRTC_RTP_TRANSCEIVER_DIRECTION_SENDONLY, caps, &trans );

But still got wrong SDP answer. In the Chrome I've got more correct answer, but playback on it fails with: dtlsconnection gstdtlsconnection.c:964:handle_error:<GstDtlsConnection@0x7fd98800d1f0> Fatal SSL error

thaytan commented 4 years ago

If you're receiving H.264 and retransmitting without reencoding, you're completely at the mercy of the IP camera and whatever H.264 profile / level it's sending you. It may or may not be usable in the browser.

thaytan commented 4 years ago

By the way, it works when you create the transceiver manually because it gets WebRTC to report a fake H.264 profile-level-id into the SDP. That works because the browser can actually decode more profiles/levels than it is willing to accept if you tell it the truth.

zingfrid commented 4 years ago

I've added rtph264depay->rth264pay step and SDP started to look completely different. My SDP has a=sendonly for both cases, Chrome SDP in the answer has both:

a=sendonly
a=recvonly

but Firefox has only: a=sendrecv Can I add anything to make it work?

thaytan commented 4 years ago

It would help to see the whole SDP. I'm not sure what you're describing.

zingfrid commented 4 years ago

Here output for Firefox:

Negotiation offer created:
v=0
o=- 7161644358177594982 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 video1
m=video 9 UDP/TLS/RTP/SAVPF 105
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:i8LhaAaYMnE9ZioBlgIRSWtwzIOrs+8V
a=ice-pwd:uzOkTnB4dEkN90iwQoQ6A05Rinu9FY4d
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:105 H264/90000
a=rtcp-fb:105 nack pli
a=recvonly
a=fmtp:105 profile-level-id=4d4028;packetization-mode=1;sprop-parameter-sets=Z01AKJWgHgCJ+WbIAAADAAgAAAMBRCA=,aO48gA==
a=ssrc:262572500 msid:user1445342556@host-708dbb5a webrtctransceiver2
a=ssrc:262572500 cname:user1445342556@host-708dbb5a
a=mid:video0
a=fingerprint:sha-256 6C:7F:4D:1E:4C:AF:93:F9:CD:4D:8F:6B:D5:63:92:03:F7:13:44:15:DB:1C:CB:2E:C6:D8:9E:B0:36:16:D3:C4
m=video 0 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:i8LhaAaYMnE9ZioBlgIRSWtwzIOrs+8V
a=ice-pwd:uzOkTnB4dEkN90iwQoQ6A05Rinu9FY4d
a=bundle-only
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=mid:video1
a=fingerprint:sha-256 6C:7F:4D:1E:4C:AF:93:F9:CD:4D:8F:6B:D5:63:92:03:F7:13:44:15:DB:1C:CB:2E:C6:D8:9E:B0:36:16:D3:C4

Received SDP:
v=0
o=mozilla...THIS_IS_SDPARTA-75.0 408960948908296999 0 IN IP4 0.0.0.0
s=-
t=0 0
a=sendrecv
a=fingerprint:sha-256 0A:C7:A2:56:7D:1F:06:82:6D:41:A0:23:A5:E8:F7:62:BC:9B:C8:66:4A:F9:B7:DA:F1:49:5F:EC:11:2E:E5:C1
a=ice-options:trickle
a=msid-semantic:WMS *
m=video 0 UDP/TLS/RTP/SAVPF 120
c=IN IP4 0.0.0.0
a=inactive
a=mid:video0
a=rtpmap:120 VP8/90000
m=video 0 UDP/TLS/RTP/SAVPF 120
c=IN IP4 0.0.0.0
a=inactive
a=mid:video1
a=rtpmap:120 VP8/90000

0:01:19.132431922 26242 0x559f9bbdf990 ERROR              webrtcbin gstwebrtcbin.c:4248:_set_description_task:<webrtcbin> media 0 is missing or contains an empty 'ice-ufrag' attribute

Here is for Chrome:

Negotiation offer created:
v=0
o=- 5985642433545516472 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE application0 video1
m=application 9 UDP/TLS/RTP/SAVPF 107
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:F+Pclsy3D9XCZrv4PLg1MRpSC6wr0309
a=ice-pwd:sizMbxHH9Y7/cH7vTgOrwuFb5oohAI4m
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:107 VND.ONVIF.METADATA/90000
a=rtcp-fb:107 nack pli
a=recvonly
a=fmtp:107 decodertag=h3c-v3 RTCP=0
a=ssrc:116046122 msid:user3302686175@host-79943653 webrtctransceiver0
a=ssrc:116046122 cname:user3302686175@host-79943653
a=mid:application0
a=fingerprint:sha-256 A8:AB:1E:47:9E:89:66:7A:AF:FD:BC:EB:44:82:CA:A3:7B:4F:8A:8E:C2:37:0D:70:C5:C6:67:F4:39:92:3F:47
m=video 0 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:F+Pclsy3D9XCZrv4PLg1MRpSC6wr0309
a=ice-pwd:sizMbxHH9Y7/cH7vTgOrwuFb5oohAI4m
a=bundle-only
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=mid:video1
a=fingerprint:sha-256 A8:AB:1E:47:9E:89:66:7A:AF:FD:BC:EB:44:82:CA:A3:7B:4F:8A:8E:C2:37:0D:70:C5:C6:67:F4:39:92:3F:47

Received SDP:
v=0
o=- 2613559437304585396 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE application0 video1
a=msid-semantic: WMS
m=application 9 UDP/TLS/RTP/SAVPF 0
c=IN IP4 0.0.0.0
b=AS:30
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Nj5k
a=ice-pwd:tDL+CSNnj7AYAvMBv27uJKss
a=ice-options:trickle
a=fingerprint:sha-256 08:B2:95:EA:77:CC:D4:07:ED:E8:16:09:8A:C2:06:23:81:56:1A:78:43:8B:04:DC:E6:2A:C9:E0:F0:93:FE:31
a=setup:active
a=mid:application0
a=sendonly
a=rtcp-mux
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:Nj5k
a=ice-pwd:tDL+CSNnj7AYAvMBv27uJKss
a=ice-options:trickle
a=fingerprint:sha-256 08:B2:95:EA:77:CC:D4:07:ED:E8:16:09:8A:C2:06:23:81:56:1A:78:43:8B:04:DC:E6:2A:C9:E0:F0:93:FE:31
a=setup:active
a=mid:video1
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=0;profile-level-id=42e01f

0:00:08.164452458 32291 0x55db874720a0 ERROR              webrtcbin gstwebrtcbin.c:4193:_update_transceivers_from_sdp:<webrtcbin> Unknown media type in SDP at index 0

Both not working

thaytan commented 4 years ago

Pretty strange SDPs. Why is the offer generated to Chrome different to the one for Firefox? The Chrome offer has incomplete caps and also includes an ONVIF metadata stream? Maybe your setup code is racy.

It's also strange that you're sending an offer with 2 video streams - 1 sendonly video stream, and 1 recvonly. In the SDP sent to Firefox, it has a recvonly video stream (asking Firefox to send a stream to us), and specifies a profile-level-id that Firefox doesn't support, so it comes back inactive. The other stream 'sendonly' doesn't contain profile-level-id, and firefox rejects it for that reason. The field is probably missing because the caps are incomplete - you might need to wait for the pipeline to preroll some data from the webcam before you can send the WebRTC offer.

zingfrid commented 4 years ago

Thank you very much for your help. Because this topic is not googlable at all. So, as you predict, I faced an issue, that profile-level-id is not supported by browsers. First of all, I was not able to find a list of supported profiles for each browser. Second, can you explain a bit, how does webrtcbin generate SDP offer. What data does influent on outgoing profile-level-id generation? For example, when I have rtspsrc->webrtcsink pipeline, I have profile-level-id=42001f. When I add rtspsrc->depay->parse->pay->webrtcsink, I have profile-level-id=64002a.

thaytan commented 4 years ago

The profile-level-id is generated from the H.264 profile and level in the codec_data or SPS. It's calculated here in the payloader: https://gitlab.freedesktop.org/gstreamer/gst-plugins-good/blob/798f320ba7b38c65cbf2069e62e072903ad0ad9d/gst/rtp/gstrtph264pay.c#L485

When you take it straight from rtspsrc, it comes from the profile-level-id of the SDP the RTSP server provides, which might not actually match the stream.

zingfrid commented 4 years ago

I understand, that with browsers there can be profile-level-id issues. So, I decided, first to try it on GST->WebRTC->GST chain. From c sendrecv code I've made sendonly, and from python one, I've made recv only. It successfully establishes a connection, gathering ice candidates. But after that, on the sender side I have an error: ERROR webrtcbin gstwebrtcbin.c:4248:_set_description_task:<sendrecv> Fingerprint in media 1 differs from previous fingerprint. sha-256 C6:72:49:E0:FF:F0:12:BD:25:98:57:2B:C2:72:AE:0F:26:DE:FD:90:38:68:4B:46:46:CB:DC:C0:22:1E:F5:38' != 'sha-256 7E:DC:FC:CC:A4:94:F2:02:9B:E3:FD:52:76:7C:FB:E8:C6:5D:BF:D1:1C:AC:76:EE:29:5C:88:C7:3A:83:F5:D6'

My Sender offer:

Sending offer:
v=0
o=- 6584277422724955141 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=group:BUNDLE video0 application1
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:xt4XLsdVoRCRlum2fPJtmRGJL7fb0ENO
a=ice-pwd:MENhziE42JlR7XeyIPaZ0P3oT/fE76E3
a=rtcp-mux
a=rtcp-rsize
a=sendonly
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=42c015;sprop-parameter-sets=Z0LAFdkBQfsBagwCDUoAAAMAAgAAAwB5HixckA==,aMuMsg==
a=ssrc:3980819698 msid:user805512454@host-5bc3a76 webrtctransceiver0
a=ssrc:3980819698 cname:user805512454@host-5bc3a76
a=mid:video0
a=fingerprint:sha-256 7E:DC:FC:CC:A4:94:F2:02:9B:E3:FD:52:76:7C:FB:E8:C6:5D:BF:D1:1C:AC:76:EE:29:5C:88:C7:3A:83:F5:D6
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:xt4XLsdVoRCRlum2fPJtmRGJL7fb0ENO
a=ice-pwd:MENhziE42JlR7XeyIPaZ0P3oT/fE76E3
a=bundle-only
a=mid:application1
a=sctp-port:5000
a=fingerprint:sha-256 7E:DC:FC:CC:A4:94:F2:02:9B:E3:FD:52:76:7C:FB:E8:C6:5D:BF:D1:1C:AC:76:EE:29:5C:88:C7:3A:83:F5:D6

Reciever's answer:

Received answer:
v=0
o=- 6584277422724955141 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=ice-ufrag:8B+c/yOc6wLYa0wmvaT0AfWV4mAvxV7Z
a=ice-pwd:LDIfvGdFQ+H2x2D9A09+Db1GpzWfpJkn
a=rtcp-mux
a=mid:video0
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 packetization-mode=1;profile-level-id=42c015;sprop-parameter-sets=Z0LAFdkBQfsBagwCDUoAAAMAAgAAAwB5HixckA==,aMuMsg==
a=recvonly
a=setup:active
a=fingerprint:sha-256 C6:72:49:E0:FF:F0:12:BD:25:98:57:2B:C2:72:AE:0F:26:DE:FD:90:38:68:4B:46:46:CB:DC:C0:22:1E:F5:38
m=application 0 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:xt4XLsdVoRCRlum2fPJtmRGJL7fb0ENO
a=ice-pwd:MENhziE42JlR7XeyIPaZ0P3oT/fE76E3
a=bundle-only
a=mid:application1
a=sctp-port:5000
a=fingerprint:sha-256 7E:DC:FC:CC:A4:94:F2:02:9B:E3:FD:52:76:7C:FB:E8:C6:5D:BF:D1:1C:AC:76:EE:29:5C:88:C7:3A:83:F5:D6

Why does it happen?

thaytan commented 4 years ago

I've never seen that. It's probably a bug or limitation in webrtcbin, but maybe triggered by something weird in your test code? The answer SDP has changed the a=fingerprint:sha-256 line for the video media compared to what the offer had

ystreet commented 4 years ago

The data channel section has been rejected. This is a known case where webrtcbin doesn't quite get the SDP negotiation correct.

thaytan commented 4 years ago

I just noticed the reply SDP in https://github.com/centricular/gstwebrtc-demos/issues/172#issuecomment-615199041 doesn't have BUNDLE in it. Is the answerer running an older version of GStreamer than the offerer?

dudulashok commented 4 years ago

Can you please help me how to use webrtc-unidirectional-h264? I'm facing the same issue as you. I built the application using the following command:

gcc webrtc-unidirectional-h264.c $(pkg-config --cflags --libs gstreamer-webrtc-1.0 gstreamer-sdp-1.0 libsoup-2.4 json-glib-1.0) -o webrtc-unidirectional-h264

Then the output is as below: WebRTC page link: http://127.0.0.1:57778/

Processing new websocket connection 0x.....

Closed websocket connection 0x....

There is no output.

zingfrid commented 4 years ago

Hello @thaytan. Thank you very much for your help. I was able to run it with gst->gst concept. It looks like that it were issues with profile-ids for browsers.

@dudulashok I was not able to run webrtc-unidirectional-h264. I've redesigned sendrecv example to be sendonly for c, and recvonly for python.

zingfrid commented 3 years ago

Hello Ali,

I've got everything working, but I was doing with C<->Python bindings with no browser involved. Unfortunately, I'm not able to help you with browser.

Best regards,

On Tue, Mar 9, 2021, 14:03 Ali @.***> wrote:

@zingfrid https://github.com/zingfrid Hey have you got an example lying around of how you got sendrecv working for receiving on the client only? For some reason the remote video won't show unless I use getUserMedia

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/centricular/gstwebrtc-demos/issues/172#issuecomment-793787899, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADAKCJQ6EHC6UJF7DR3RQS3TCYFB5ANCNFSM4MEAQ7AA .

ystreet commented 3 years ago

These examples have moved to https://gitlab.freedesktop.org/gstreamer/gst-examples/-/tree/master/webrtc and this repository will not be updated further.