Add webrtc support - Githubissues

fnoop commented 4 years ago

Notes here: https://github.com/goodrobots/maverick/issues/693

fnoop commented 4 years ago

Initial webrtc support should use the simplest/easiest approach first:

Use Peer Connection, not Data Channels
Use fixed encoding (whatever is most widely used in browsers - h264/vp8)
Assume local network usage with minimal signalling, so no NAT traversal support yet.
Simple p2p/mesh model, no SFU or other more complex topologies

fnoop commented 4 years ago

Use adapter.js to hide browser quirks: https://github.com/webrtc/adapter

fnoop commented 4 years ago

Possible alternative to gstreamer: https://github.com/aiortc/aiortc

fnoop commented 4 years ago

SSL issues: https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/issues/811 Commenting out MinProtocol in /etc/ssl/openssl.cnf works, but probably not a good solution.

fnoop commented 4 years ago

Create simple p2p model initially as an output module of visiond, with a simple signalling server based on concepts here: https://github.com/centricular/gstwebrtc-demos/

SamuelDudley commented 4 years ago

latency hints: https://github.com/centricular/gstwebrtc-demos/issues/102#issuecomment-483329064

SamuelDudley commented 4 years ago

Having a data channel in parallel with video would be useful for HUD overlays client side / other metadata

https://github.com/aiortc/aiortc/tree/master/examples/server

Note all the aiortc stuff uses opencv. It would be interesting to know if we could plug the gstreamer pipeline into aiortc as a data source.

fnoop commented 4 years ago

So I've been thinking about this for a while (HUD overlays). The problem with doing the overlays server (CC) side is:

Difficult: it's hard to do cross-platform graphic overlays. Some projects use hardware GL, some use the GPU to accelerate the computation, some do it in software (v.slow). Anything hardware/gpu based needs a complex toolchain specific to the platform.
Slow and poor quality: you generally have to use opencv for cross-platform graphic overlays. Opencv is very slow (cpu-bound) and poor quality - the vector and text drawing is pretty ugly.
Lack of flexibility/interactivity - v.difficult for the user to interactive and turn on/off, choose different display modes, data points etc

So the HUD mode of maverick-web was always intended to overlay on top of FPV. It uses hardware accelerated vector drawing libraries within the client/user browser to draw the overlays, which is generally much faster, prettier, and much more flexible. The downside is going to be coordinating the correct data in time with the video - either we do this through timestamps, or more likely to start with we just add a configurable delay into the -web config. Web browsers are the single most mature, advanced, developed piece of software on the desktop these days and are such a rich development environment, we should take advantage of what they have to offer, imo, and do as much as we can in them.

SamuelDudley commented 4 years ago

Yeah that's where my thinking was going. We could embed a minimal amount of data in the webrtc stream and pull it out to help draw the HUD. Really we could do away with most of the metadata, but leave in some sort of timestamp metadata. The timestamp can be used to sync the video to a seperate mavlink stream.

SamuelDudley commented 4 years ago

Just dumping some links for reading

https://devtalk.nvidia.com/default/topic/1056918/jetson-tx2/nvarguscamerasrc-buffer-metadata-is-missing-/post/5358980/#5358980

http://gstreamer-devel.966125.n4.nabble.com/WebRTC-with-frame-synchronized-metadata-td4689307.html

https://github.com/phongcao/webrtc-metadata-example

https://github.com/mpromonet/webrtc-streamer/

fnoop commented 4 years ago

Hmm where should we put/look for the certs, and should we autogenerate them if they don't exist?

SamuelDudley commented 4 years ago

We can set the path in the configuration, but if not found they should be automatically generated imo. Not fussed re path, perhaps near the config file?

cglusky commented 4 years ago

what is the plan for authentication in general? anything else requiring certs up the stack? and can certs be used more globally as required?

fnoop commented 4 years ago

There's already a basic SSL setup in Maverick - it creates a CA in ~/data/security/ssl, then various services (ldap, web) create extra certs as required. There's a disabled webpage in -web-legacy that checks if the user browser has the self-signed cert installed and if not prompts on how to install it. It needs consolidating and tidying up, documenting and presenting to the user properly. SSL is a requirement for webrtc, and there's no actual way of creating recognised CA signed certs even if we wanted to, because the hostnames/IPs that CCs use are not public. So we have to create a self-signed CA, and find some way for users to consume it.

For maverick, I'll add an exec in the visiond manifest that generates the certs and bungs them in ~/data/security/ssl/visiond, and alter the visiond.conf (specific to maverick) to point to there. For standalone visiond we can provide instructions in the readme.

fnoop commented 4 years ago

https://github.com/goodrobots/maverick/issues/921

fnoop commented 4 years ago

Feb 24 17:14:33 maverick-raspberry python3[3201]: 2020-02-24 17:14:33,560 - visiond.modules.webrtc - INFO - Starting peer connection with signalling server
Feb 24 17:14:33 maverick-raspberry python3[3201]: 2020-02-24 17:14:33,564 - visiond.modules.webrtc - WARNING - connect_loop error: OSError("Multiple exceptions: [Errno 111] Connect call failed ('::1', 8443, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 8443)")
Feb 24 17:14:34 maverick-raspberry python3[3201]: 2020-02-24 17:14:34,566 - visiond.modules.webrtc - INFO - Starting peer connection with signalling server
Feb 24 17:14:34 maverick-raspberry python3[3201]: 2020-02-24 17:14:34,571 - visiond.modules.webrtc - WARNING - connect_loop error: OSError("Multiple exceptions: [Errno 111] Connect call failed ('::1', 8443, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 8443)")
Feb 24 17:14:35 maverick-raspberry python3[3201]: 2020-02-24 17:14:35,572 - visiond.modules.webrtc - INFO - Starting peer connection with signalling server
Feb 24 17:14:35 maverick-raspberry python3[3201]: 2020-02-24 17:14:35,576 - visiond.modules.webrtc - WARNING - connect_loop error: OSError("Multiple exceptions: [Errno 111] Connect call failed ('::1', 8443, 0, 0), [Errno 111] Connect call failed ('127.0.0.1', 8443)")

Need to limit the number of times this retries - at the moment if there is a fault starting up the signalling server this just retries forever.

fnoop commented 4 years ago

Major bug in gstreamer/python means visiond won't support send-only webrtc streaming on raspberry/tegra which provide somewhat proprietary versions of gstreamer 1.14: centricular/gstwebrtc-demos#144

In the short term, we'll have to look at aiortc or janus or something similar as an alternative to get us going.

All other platforms we custom compile the latest gstreamer so we can continue long-term work to get this working with visiond.

goodrobots / visiond

Add webrtc support #7