search

dusty-nv / jetson-inference

Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson.
https://developer.nvidia.com/embedded/twodaystoademo
MIT License
7.86k stars 2.98k forks source link

WebRTC Server issues #1802

Closed pq53ui closed 8 months ago

pq53ui commented 8 months ago

I haven't been able to find a similar issue to what I am experiencing.

Running either detectnet, segnet, or my custom script I am unable to reliably connect to the webrtc output in browser. I followed the tutorial and tried both the secure version with SSL keys and no keys.

When I run the script, while the models and streams are initializing I open the browser and access the output stream. Sometimes it is successful sometimes all I get is a gray screen with video trying to load.

Screenshot from 2024-03-02 17-54-10

So, it seems like there is some sweet spot while I can access the stream, but I have no idea when exactly that is. Sometime between initialization. While it is in the image capture loop, it is impossible to see any output.

** This is the output when I run the script and connect to the stream. I removed the [TRT] logs and others that have nothing to do with webrtc or gstreamer in my opinion. I don't know much about web related things, so any help would be much appreciated. 


[gstreamer] gstCamera successfully created device v4l2:///dev/video0
[video]  created gstCamera from v4l2:///dev/video0
------------------------------------------------
gstCamera video options:
------------------------------------------------
  -- URI: v4l2:///dev/video0
     - protocol:  v4l2
     - location:  /dev/video0
  -- deviceType: v4l2
  -- ioType:     input
  -- codec:      raw
  -- codecType:  cpu
  -- width:      640
  -- height:     360
  -- frameRate:  30
  -- numBuffers: 4
  -- zeroCopy:   true
  -- flipMethod: none
------------------------------------------------
[gstreamer] gstEncoder -- codec not specified, defaulting to H.264
[gstreamer] gstEncoder -- pipeline launch string:
[gstreamer] appsrc name=mysource is-live=true do-timestamp=true format=3 ! omxh264enc name=encoder bitrate=4000000 insert-sps-pps=1 insert-vui=1 ! video/x-h264 ! rtph264pay config-interval=1 ! application/x-rtp,media=video,encoding-name=H264,clock-rate=90000,payload=96 ! tee name=videotee ! queue ! fakesink
[webrtc] WebRTC server started @ http://jetson-desktop:8554
[webrtc] websocket route added /output
[video]  created gstEncoder from webrtc://@:8554/output
------------------------------------------------
gstEncoder video options:
------------------------------------------------
  -- URI: webrtc://@:8554/output
     - protocol:  webrtc
     - location:  0.0.0.0
     - port:      8554
  -- deviceType: ip
  -- ioType:     output
  -- codec:      H264
  -- codecType:  omx
[webrtc] WebRTC server thread running...
  -- width:      640
  -- height:     360
  -- frameRate:  30
  -- bitRate:    4000000
  -- numBuffers: 4
  -- zeroCopy:   true
  -- latency     10
------------------------------------------------

[webrtc] HTTP 192.168.1.5 GET '/'
[webrtc] /
[webrtc] HTTP 192.168.1.5 GET '/favicon.ico'
[webrtc] /favicon.ico
[webrtc] websocket /output -- new connection opened by 192.168.1.5 (peer_id=0)
[webrtc] new WebRTC peer connecting (192.168.1.5, peer_id=0)

(python3:28817): GLib-GObject-WARNING **: 18:04:54.081: g_object_set_is_valid_property: object class 'GstWebRTCBin' has no property named 'latency'

(python3:28817): GLib-GObject-WARNING **: 18:04:54.082: g_object_set_is_valid_property: object class 'WebRTCTransceiver' has no property named 'direction'

[webrtc] negotiation offer created:
v=0
o=- 8241377557727837600 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=msid-semantic:WMS webrtcbin-0
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:D6jux9GBpx/BE7rpvjM3S6oW+mu11ykl
a=ice-pwd:RjC+Oz5ejsks1F82f8rgaUFiFcw5K+gg
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=framerate:30
a=fmtp:96 packetization-mode=1;profile-level-id=424015;sprop-parameter-sets=Z0JAKJWgKA9oQAAAAwBAAAAPI8IhGoA=,aM48gA==
a=ssrc:2128262644 msid:user390821661@host-e84f3e36 webrtctransceiver0
a=ssrc:2128262644 cname:user390821661@host-e84f3e36
a=mid:video0
a=fingerprint:sha-256 77:AD:8B:1B:CF:77:00:CA:43:4F:4A:C8:B2:73:F9:37:73:82:24:A0:D2:E0:AE:6E:05:7B:6A:AA:1D:9C:14:ED

[webrtc] sending offer for /output to 192.168.1.5 (peer_id=0): 
{"type":"sdp","data":{"type":"offer","sdp":"v=0\r\no=- 8241377557727837600 0 IN IP4 0.0.0.0\r\ns=-\r\nt=0 0\r\na=ice-options:trickle\r\na=msid-semantic:WMS webrtcbin-0\r\nm=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\na=setup:actpass\r\na=ice-ufrag:D6jux9GBpx/BE7rpvjM3S6oW+mu11ykl\r\na=ice-pwd:RjC+Oz5ejsks1F82f8rgaUFiFcw5K+gg\r\na=sendrecv\r\na=rtcp-mux\r\na=rtcp-rsize\r\na=rtpmap:96 H264/90000\r\na=rtcp-fb:96 nack pli\r\na=framerate:30\r\na=fmtp:96 packetization-mode=1;profile-level-id=424015;sprop-parameter-sets=Z0JAKJWgKA9oQAAAAwBAAAAPI8IhGoA=,aM48gA==\r\na=ssrc:2128262644 msid:user390821661@host-e84f3e36 webrtctransceiver0\r\na=ssrc:2128262644 cname:user390821661@host-e84f3e36\r\na=mid:video0\r\na=fingerprint:sha-256 77:AD:8B:1B:CF:77:00:CA:43:4F:4A:C8:B2:73:F9:37:73:82:24:A0:D2:E0:AE:6E:05:7B:6A:AA:1D:9C:14:ED\r\n"}}
[webrtc] websocket /output -- recieved message from 192.168.1.5 (peer_id=0) (639 bytes)
[webrtc] received SDP message for /output from 192.168.1.5 (peer_id=0)
v=0
o=- 2523303434956612631 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:tp+0
a=ice-pwd:eAw6M45ERrwB9PxVZygBpptF
a=ice-options:trickle
a=fingerprint:sha-256 2E:18:50:4A:2C:CB:76:2B:7B:26:A5:9A:08:6C:DD:02:40:50:94:77:E9:2E:65:EB:9C:79:77:79:2A:61:D2:5A
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e015

[webrtc] websocket /output -- recieved message from 192.168.1.5 (peer_id=0) (236 bytes)
[webrtc] received ICE message on /output from 192.168.1.5 (peer_id=0) with mline index 0; candidate: 
candidate:1263540956 1 udp 2113937151 6593b3ef-688b-4aea-a0e7-ec8281fd24bd.local 44018 typ host generation 0 ufrag tp+0 network-cost 999
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:1 1 UDP 2013266431 fe80::f31:f242:cf6:4af4 37118 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:2 1 TCP 1015023615 fe80::f31:f242:cf6:4af4 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:3 1 TCP 1010829311 fe80::f31:f242:cf6:4af4 41869 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:4 1 UDP 2013266430 2a00:ee2:900:6500:aded:faa2:b363:ee26 57486 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:5 1 TCP 1015022079 2a00:ee2:900:6500:aded:faa2:b363:ee26 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:6 1 TCP 1010827775 2a00:ee2:900:6500:aded:faa2:b363:ee26 48111 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:7 1 UDP 2013266429 2a00:ee2:900:6500:5d99:bf18:b020:7b34 40080 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:8 1 TCP 1015022335 2a00:ee2:900:6500:5d99:bf18:b020:7b34 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:9 1 TCP 1010828031 2a00:ee2:900:6500:5d99:bf18:b020:7b34 37811 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:10 1 UDP 2013266428 172.17.0.1 57809 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:11 1 TCP 1015022591 172.17.0.1 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:12 1 TCP 1010828287 172.17.0.1 59439 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:13 1 UDP 2013266427 192.168.1.4 45851 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:14 1 TCP 1015022847 192.168.1.4 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:15 1 TCP 1010828543 192.168.1.4 45491 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:1 2 UDP 2013266430 fe80::f31:f242:cf6:4af4 32843 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:2 2 TCP 1015023614 fe80::f31:f242:cf6:4af4 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:3 2 TCP 1010829310 fe80::f31:f242:cf6:4af4 58813 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:4 2 UDP 2013266429 2a00:ee2:900:6500:aded:faa2:b363:ee26 54258 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:5 2 TCP 1015022078 2a00:ee2:900:6500:aded:faa2:b363:ee26 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:6 2 TCP 1010827774 2a00:ee2:900:6500:aded:faa2:b363:ee26 36529 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:7 2 UDP 2013266428 2a00:ee2:900:6500:5d99:bf18:b020:7b34 50122 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:8 2 TCP 1015022334 2a00:ee2:900:6500:5d99:bf18:b020:7b34 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:9 2 TCP 1010828030 2a00:ee2:900:6500:5d99:bf18:b020:7b34 38667 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:10 2 UDP 2013266427 172.17.0.1 56839 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:11 2 TCP 1015022590 172.17.0.1 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:12 2 TCP 1010828286 172.17.0.1 48383 typ host tcptype passive"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:13 2 UDP 2013266426 192.168.1.4 53093 typ host"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:14 2 TCP 1015022846 192.168.1.4 9 typ host tcptype active"}}
[webrtc] sending ICE candidate for /output to 192.168.1.5 (peer_id=0): 
{"type":"ice","data":{"sdpMLineIndex":0,"candidate":"candidate:15 2 TCP 1010828542 192.168.1.4 56675 typ host tcptype passive"}}
[webrtc] resolved 6593b3ef-688b-4aea-a0e7-ec8281fd24bd.local for 192.168.1.5 in incoming ICE message
[webrtc] candidate:1263540956 1 udp 2113937151 192.168.1.5 44018 typ host generation 0 ufrag tp+0 network-cost 999
[webrtc] websocket /output -- recieved message from 192.168.1.5 (peer_id=0) (236 bytes)
[webrtc] received ICE message on /output from 192.168.1.5 (peer_id=0) with mline index 0; candidate: 
candidate:3086921863 1 udp 2113939711 40925334-02af-46e9-9d8a-9b1b36f8da7d.local 49420 typ host generation 0 ufrag tp+0 network-cost 999
[gstreamer] gstreamer stream status CREATE ==> src
dusty-nv commented 8 months ago

Hi @pq53ui, are you able to view WebRTC stream with video-viewer / video-viewer.py first?

It looks like the mDNS hostname resolution is working for you, but have you also tried setting chrome://flags/#enable-webrtc-hide-local-ips-with-mdns in your browser?

pq53ui commented 8 months ago

Thanks for the quick response.

I ran the video-viewer.py Same issue, I get the same gray screen as in the screenshot. I have tried enabling the chrome flag as well, no difference.

I am getting the same output as I have posted above

Maybe a few additional outputs

[webrtc] resolved 5ff934f4-6ebf-4a7c-984c-b4a83b1134df.local for 192.168.1.5 in incoming ICE message
[webrtc] candidate:2191427754 1 udp 2113937151 192.168.1.5 57232 typ host generation 0 ufrag l8h+ network-cost 999
[webrtc] websocket /output -- recieved message from 192.168.1.5 (peer_id=0) (236 bytes)
[webrtc] received ICE message on /output from 192.168.1.5 (peer_id=0) with mline index 0; candidate: 
candidate:2128326268 1 udp 2113939711 9aed251a-ffef-4d2c-a1bd-845889b8c5e0.local 54438 typ host generation 0 ufrag l8h+ network-cost 999
[gstreamer] gstEncoder -- pipeline full, skipping frame 25 (640x360, 345600 bytes)
video-viewer:  captured 25 frames (640 x 360)

I assume this is due to not running it in --headless. If I run it in --headless, I don't get the pipeline full warnings.

I have also seen 

[network] getHostByName() trying to resolve host '9aed251a-ffef-4d2c-a1bd-845889b8c5e0.local' (retry 1 of 10)

this line a few times before. Trying to understand if that plays any role as well.

Its bugging me that it sometimes works under certain (unknown) conditions. But in most cases I can't get it to work.

dusty-nv commented 8 months ago

I have also seen

[network] getHostByName() trying to resolve host '9aed251a-ffef-4d2c-a1bd-845889b8c5e0.local' (retry 1 of 10)

this line a few times before. Trying to understand if that plays any role as well.

Ok yea, that is the mDNS thing - if you disable that chrome flag and restart the browser, you shouldn't get that anymore

[gstreamer] gstEncoder -- pipeline full, skipping frame 25 (640x360, 345600 bytes)

Try commenting out this if block of code, and recompiling/reinstalling:

https://github.com/dusty-nv/jetson-utils/blob/5a5bb422eada7fdbd7d0f53806cf8e0889f2352f/codec/gstEncoder.cpp#L515

Sorry that WebRTC can be finicky sometimes with the connection/negotiation, and the gstreamer plugins for it aren't super robust

pq53ui commented 8 months ago

Disabling that chrome flag helped. I am able to see the stream and reconnect to it while the capture loop is going too!

Thank you! I am mostly working in headless mode, so pipeline full doesn't worry me too much right now. But it's good to know how to make it disappear.

And again, thanks for quick response.

dgdgksj commented 7 months ago

I have also seen

[network] getHostByName() trying to resolve host '9aed251a-ffef-4d2c-a1bd-845889b8c5e0.local' (retry 1 of 10)

this line a few times before. Trying to understand if that plays any role as well.

Ok yea, that is the mDNS thing - if you disable that chrome flag and restart the browser, you shouldn't get that anymore

[gstreamer] gstEncoder -- pipeline full, skipping frame 25 (640x360, 345600 bytes)

Try commenting out this if block of code, and recompiling/reinstalling:

https://github.com/dusty-nv/jetson-utils/blob/5a5bb422eada7fdbd7d0f53806cf8e0889f2352f/codec/gstEncoder.cpp#L515

Sorry that WebRTC can be finicky sometimes with the connection/negotiation, and the gstreamer plugins for it aren't super robust

Thanks @dusty-nv Just by disabling the "Anonymize local IPs exposed by WebRTC" option in chrome://flags/, I got it resolved.