search

goodrobots / maverick

UAV Autonomous Systems Management
https://goodrobots.github.io/maverick/
MIT License
171 stars 61 forks source link

Add web based realtime video #693

Open fnoop opened 6 years ago

fnoop commented 6 years ago

https://developer.mozilla.org/en-US/Apps/Fundamentals/Audio_and_video_delivery/Live_streaming_web_audio_and_video

fnoop commented 6 years ago

https://github.com/video-dev/hls.js http://videojs.com/ https://www.isrv.pw/html5-live-streaming-with-mpeg-dash https://github.com/arut/nginx-rtmp-module http://nginx-rtmp.blogspot.co.uk/ https://coaxion.net/blog/2013/10/streaming-gstreamer-pipelines-via-http/ https://coaxion.net/blog/2014/05/http-adaptive-streaming-with-gstreamer/ https://github.com/131/h264-live-player https://github.com/mbebenita/Broadway

fnoop commented 6 years ago

Hls/Dash features chunked playlist video streaming with rich javascript frontends. Unfortunately it looks like hls/dash are inherently unsuitable for realtime video, and are far more complex to implement. Initially, implement a much simpler http video just to get something working. h264 in javascript looks promising.

fnoop commented 6 years ago

RTSP plugin for chrome works well with visiond: https://www.videoexpertsgroup.com/vxg-chrome-plugin/

fnoop commented 6 years ago

Looks like https://github.com/131/h264-live-player is likely to be along the right route, but needs websockets and server implementation. Therefore bumping this to Maverick 1.2 target and will look to implement in web ui. For now, add a simple page with the vxg-chrome-plugin support.

fnoop commented 6 years ago

Simple rtsp plugin video added, re-milestoning to 1.2 for non-plugin video

SamuelDudley commented 6 years ago

@fnoop For what its worth, I have also had a look in this space previously and ended up here: https://github.com/131/h264-live-player. What are your thoughts on running the JS server on the platform to stream video?

fnoop commented 6 years ago

@SamuelDudley Yes I looked at that, but from a quick glance it looks like it converts the rtsp/udp stream to websockets/tcp at the server end. The advantage of the vxg method is that it does that conversion at the browser end, so the stream between the uav and the ground browser remains udp (at least that's what it looked like at first glance). This one uses a similar method: https://github.com/Streamedian/html5_rtsp_player/wiki/HTML5-RTSP-Player So none of these are actually using rtsp between the server and the client except for the vxg method, but rather are transmuting or encapsulating in websockets (to my limited understanding). Looked at webrtc as well but that's more complex at the server end and not supported by ios. Suspect that when we look at the webui proper, we'll end up implementing the websockets proxy method, unless we can come up with way of doing it on the browser in javascript or MSE.

cglusky commented 6 years ago

I think it's spot on to offer easy to use video preview.

Wish this was open source: https://www.linux-projects.org/uv4l/

Also suggest this come with a caution to the user. It would only take a few client connects to saturate your real world WiFi link if offering a 720 or 1080 stream. 4G connect might be a bit better than wifi but I bet it would not take much to trip QoS upload limits for most service providers. I have no hard data handy but there are calculators like this one: http://www.stardot.com/bandwidth-and-storage-calculator

For GCS env it would might be good to have roadmap for ways to handle sharing?

https://janus.conf.meetecho.com/index.html https://jitsi.org/

fnoop commented 5 years ago

Good description of webrtc session: https://levelup.gitconnected.com/build-your-own-video-chat-with-vue-webrtc-socketio-node-redis-eb51b78f9f55

fnoop commented 4 years ago

Implement WebRTC. Good support now been merged into gstreamer: https://opensource.com/article/19/1/gstreamer https://github.com/centricular/gstwebrtc-demos Note webrtc on gstreamer has funky features like FEC that made a big difference in wifibroadcast. https://gstreamer.freedesktop.org/documentation/webrtc/index.html?gi-language=c https://gstreamer.freedesktop.org/data/events/gstreamer-conference/2017/Matthew%20Waters%20-%20GStreamer%20WebRTC.pdf http://blog.nirbheek.in/2018/02/gstreamer-webrtc.html

cglusky commented 4 years ago

So is this going to be LAN only. No need for STUN or TURN servers whilst handling the signaling locally as described in the vue demo link you provided back in August? I think that works if you only want one way video with option to go to a relay server for those who want to get it out on WAN. But WebRTC makes my head hurt.

fnoop commented 4 years ago

WebRTC is peer to peer, very low latency. There are no intermediate/proxy servers. It also allows either uni or bi-directional video along with a host of other features. I think it's routable so can be either 'LAN' or 'WAN' - although not sure those distinctions are relevant for this case. It's a real headache to understand - unnecessarily complicated - but once we get a recipe worked out then I suspect this will be the 'ultimate' solution. Hardware accelerated support is also built-in to pretty much every desktop and mobile browser as well by now.

SamuelDudley commented 4 years ago

My understanding is that the data communication is peer to peer but the STUN / TURN servers are needed for discovery and messaging.

https://github.com/centricular/gstwebrtc-demos/blob/master/sendrecv/gst/webrtc-sendrecv.py might be a good place to start

cglusky commented 4 years ago

from what i understand if we keep it to LAN it's not nearly as bad. yes, STUN/TURN required to move out to WAN and traverse firewall/NAT.

also read that webRTC is complicated because companies make money providing STUN/TURN services. someday perhaps we can do a dApp but don't think we need to run our own private blockchain just yet.

SamuelDudley commented 4 years ago

https://stackoverflow.com/questions/45613981/is-webrtc-without-any-server-not-even-a-signaling-server-possible

cglusky commented 4 years ago

Good description of webrtc session: https://levelup.gitconnected.com/build-your-own-video-chat-with-vue-webrtc-socketio-node-redis-eb51b78f9f55

provides signaling example in that demo.

fnoop commented 4 years ago

Oh thank god, sample python code. I can't tell you how painful working out gst rtsp python code was in visiond..

cglusky commented 4 years ago

and we may want to support full webRTC in future as one of the features of dronesense and other similar platforms is to allow text chat to help teams comm over distance

fnoop commented 4 years ago

I'm hoping we can use webrtc to relay video as well - kind of like meshed networking. An obvious application would be where you have a 'mothership' higher-altitude relay UAV like a circling long-endurance plane, that a bunch of other UAVs connect to for control, video and telemetry relay. The mothership would then act as a 'server' that clients on the ground can connect to, to retrieve/display the video and telemetry, and relay control commands to the UAVs.

SamuelDudley commented 4 years ago

The hen / chick model you mention is important for SAR and is pretty high up my list of interests.

I guess first step is to get the example python code running...

fnoop commented 4 years ago

Added webrtcbin in https://github.com/goodrobots/maverick/issues/906

cglusky commented 4 years ago

This will only work maverick to maverick correct? Just based on fact you will need to roll your own signaling so you need same client both sides.

fnoop commented 4 years ago

Kind of - as I understand it there is no 'standard' webrtc client. But we should be able to make literally a single file webpage that could be served anywhere - even publicly - that people could use to connect to the backends.

fnoop commented 4 years ago

Optimistically moving to 1.2 milestone

fnoop commented 4 years ago

gstreamer webrtcbin is now available:

[dev] [mav@maverick-ubuntuvm ~/code/vid]$ gst-inspect-1.0 webrtcbin
Factory Details:
  Rank                     primary (256)
  Long-name                WebRTC Bin
  Klass                    Filter/Network/WebRTC
  Description              A bin for webrtc connections
  Author                   Matthew Waters <matthew@centricular.com>

Plugin Details:
  Name                     webrtc
  Description              WebRTC plugins
  Filename                 /srv/maverick/software/gstreamer/lib/gstreamer-1.0/libgstwebrtc.so
  Version                  1.16.2
  License                  LGPL
  Source module            gst-plugins-bad
  Source release date      2019-12-03
  Binary package           GStreamer Bad Plug-ins source release
  Origin URL               Unknown package origin

However python gobject introspection can't find the typelibs:

[dev] [mav@maverick-ubuntuvm ~/code/vid]$ python3 webrtc-sendrecv.py
Traceback (most recent call last):
  File "webrtc-sendrecv.py", line 11, in <module>
    gi.require_version('Gst', '1.0')
  File "/srv/maverick/software/python/lib/python3.7/site-packages/gi/__init__.py", line 129, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace Gst not available

On binary installs (eg. raspberry or tegra platform), webrtcbin is available but the typelibs aren't:

[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
Traceback (most recent call last):
  File "webrtc-sendrecv.py", line 13, in <module>
    gi.require_version('GstWebRTC', '1.0')
  File "/srv/maverick/software/python/lib/python3.7/site-packages/gi/__init__.py", line 129, in require_version
    raise ValueError('Namespace %s not available' % namespace)
ValueError: Namespace GstWebRTC not available
fnoop commented 4 years ago

Looks like we need to re-compile plugins_bad (which contains webrtc) even on binary platforms as they don't seem to include the typelib. Either that or copy/hack the typelib GstWebRTC-1.0.typelib into place manually on the binary platforms, if that works. eg. on jetson nano the typelibs from binary packages live in /usr/lib/aarch64-linux-gnu/girepository-1.0. Copy GstWebRTC-1.0.typelib from a source compiled platform (maverick-ubuntuvm) into here and try.

[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
Missing gstreamer plugins: ['nice']

It does indeed seem to work, so perhaps that's an easier method, even if it's not for the same version (it's from compiled version 1.16.2, nano binary install is 1.14.5).

fnoop commented 4 years ago

Binary package gir1.2-gst-plugins-bad-1.0 provides the GSTWebRTC introspection, gstreamer1.0-nice provides the nice plugin.

[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py
usage: webrtc-sendrecv.py [-h] [--server SERVER] peerid
webrtc-sendrecv.py: error: the following arguments are required: peerid

progress!

fnoop commented 4 years ago

Using the server https://webrtc.nirbheek.in/, we can connect and negotiate successfully using webrtc-sendrecv.py:

[dev] [mav@maverick-nano ~/code/vid]$ python3 webrtc-sendrecv.py 7463
Sending offer:
v=0
o=- 2335782948630536004 0 IN IP4 0.0.0.0
s=-
t=0 0
a=ice-options:trickle
a=msid-semantic:WMS sendrecv
m=video 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:g9kMX3yktuHT343et1SaD9OweDklEMa3
a=ice-pwd:0bsj7Jr5oUZh1QRTS5B6er8Ey7o496nV
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:97 VP8/90000
a=rtcp-fb:97 nack pli
a=framerate:30
a=ssrc:1109946848 msid:user332872450@host-d718308c webrtctransceiver0
a=ssrc:1109946848 cname:user332872450@host-d718308c
a=mid:video0
a=fingerprint:sha-256 96:6A:99:BA:1C:6B:40:DE:DE:6C:C8:76:3C:0F:E9:4B:C8:B2:C2:57:1C:73:E7:A4:E4:B7:EC:75:17:D3:80:FB
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=setup:actpass
a=ice-ufrag:s3npkgATtq0mylghjmrQjZHqBNCYimFi
a=ice-pwd:Zj084uEQklS0Om2aDNCjIjqYkW42muDy
a=sendrecv
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 OPUS/48000/2
a=rtcp-fb:96 nack pli
a=fmtp:96 sprop-maxcapturerate=48000;sprop-stereo=0
a=ssrc:117495130 msid:user332872450@host-d718308c webrtctransceiver1
a=ssrc:117495130 cname:user332872450@host-d718308c
a=mid:audio1
a=fingerprint:sha-256 96:6A:99:BA:1C:6B:40:DE:DE:6C:C8:76:3C:0F:E9:4B:C8:B2:C2:57:1C:73:E7:A4:E4:B7:EC:75:17:D3:80:FB

Received answer:
v=0
o=- 8882333815724943527 2 IN IP4 127.0.0.1
s=-
t=0 0
a=msid-semantic: WMS
m=video 9 UDP/TLS/RTP/SAVPF 97
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:VDxi
a=ice-pwd:H+K/CGZ/tW7I9X9cgJ3AzLTp
a=ice-options:trickle
a=fingerprint:sha-256 3F:AF:AB:3E:52:BC:13:51:29:47:50:69:83:56:2E:01:32:E1:20:4C:C0:17:5B:E2:E9:41:C7:90:73:90:B5:46
a=setup:active
a=mid:video0
a=recvonly
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:97 VP8/90000
a=rtcp-fb:97 nack pli
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:4aCx
a=ice-pwd:q4z/ljCBqd07mfLKlv9ukXcw
a=ice-options:trickle
a=fingerprint:sha-256 3F:AF:AB:3E:52:BC:13:51:29:47:50:69:83:56:2E:01:32:E1:20:4C:C0:17:5B:E2:E9:41:C7:90:73:90:B5:46
a=setup:active
a=mid:audio1
a=recvonly
a=rtcp-mux
a=rtpmap:96 OPUS/48000/2
a=fmtp:96 minptime=10;useinbandfec=1

So now need to work out how to serve video locally through a webpage.

fnoop commented 4 years ago

Switched to use Janus-gateway as webrtc proxy in the short-medium term. This acts as a media proxy from gstreamer stream and also takes care of all the signalling. Janus provides a bunch of sample javascript for the browser signalling, or we can use a webpack node module like: https://github.com/TechTeamer/janus-api Here is an example of how to consume it through vue.js: https://gist.github.com/Informatic/fc318c7ce58b33975356183800456e7d This is worth trying to start with as it's a cleaner design.

fnoop commented 4 years ago

Adding janus in separate issue: https://github.com/goodrobots/maverick/issues/922

fnoop commented 4 years ago

Did a deep dive on gstreamer/ in visiond. 'Fixed' (bodged) rtsp server support so it works for multiple clients from a shared pipeline and also allows (all external) clients to close and reopen the stream, which was previously broken. Changed visiond to default to rtsp, and changed janus webrtc proxy in Maverick to connect to visiond rtsp by default as a client. This means we can offer visiond as a no-config (or very-little-config) service that automatically provides streaming video for any rtsp client (eg. VLC, GCS like MissonPlanner/QGC), and simultaneously also provides webrtc streaming video in a browser, which was the aim all along. Still some things to improved:

Subject to testing, we have a good video solution for 1.2 release.