Enable webrtc driver camera streaming

[marshallmassengill]

I'm opening this as an issue because... quite frankly, someone needs to make this easy for teams and I think it's you all.

Please, enable a webrtc stream for a driver camera. This isn't fancy machine learning or anything to do with Apriltags. Just streaming a camera feed over FRC-allowed ports and doing it in a web browser over UDP. We've been playing with https://github.com/bluenviron/mediamtx/ and it's just one implementation but we're cooking with it. 100-120ms latency and bandwidth consumption around 2mbps with 640x480. That app has support for the pi through libcamera but it seems to also work with ffmpeg streams. It's worth a look.

[mcm001]

The main blocker with webRTC and adjacent topics is just a knowledge gap with gstreamer (and other related streaming) libraries. I know teams have done one off things before, but having to work with a million different platforms is a bit annoying. And dealing with packaging native libraries has a high enough activation energy I haven't gotten around to it.

That said, h264 webRTC would be epic if we can figure out disabling SSL and the JNI situation. There's been some Webrtc discussion in the FRC discord by people smarter than I.

As always, PRs (or, barring that, a MVP I can copy paste) are always accepted! And if there's a feature you want, the fastest way to see it happen is to do one of those things ;).

[mcm001]

Also I'm pretty active on github coz of phone pings, but Discord might be a better platform for online chats

[mcm001]

Looks like this might Just Work if we compile OpenCV with the WITH_GSTREAMER flag? That'd be epic. We'd have to ship a custom opencv though, but that's not too hard?

[marshallmassengill]

We got mediamtx working with this config. The app is written in Go though so fair warning but the config file is just yml: https://gist.github.com/marshallmassengill/8f40a5553df6221a12f44da6f064bd74

I also started by tinkering with the python implementation aiortc (and aioice) https://github.com/marshallmassengill/aiortc. I had to fork them both for the port restrictions but that lead to a dead end involving the legacy camera stack.

No need for SSL though, in fact, STUN can be disabled this is all on a single subnet.

I'm sadly not super active on discord but if I find anything more, I'll share it. Trust me when I tell you that this is a game changer for teams in terms of a driver camera experience. It's miles better than rtsp and much more performant from what we've seen. I do agree that the multitude of platforms might make this more complex though.

[mcm001]

I've just pulled their docker container and got their python code working with my webcam, which is fun. Thanks for the yaml config, that's super helpful. Seeing about a second of lag right now with their defaults

[mcm001]

Got it working (by some definition of "working") in a unit test!

[mcm001]

100% a skill issue on my part. The frame size I gave when constructing the VideoWriter and the frame size I was giving it didn't match.

[mcm001]

[marshallmassengill]

Now that's slick.

[mcm001]

For any future travelers: still looking for a motivated contributor to push this through!

[philltran]

I am interested in helping. Hopefully6 I will have some time over Christmas break.

[mcm001]

Awesome!

Not sure if mediamtx is the best way to do this? There are other options. I just used it since it was suggested to me, but a quick google shows some other golang results like https://github.com/deepch/RTSPtoWeb/blob/master/docs/examples/webrtc/main.js

[mcm001]

And code: https://github.com/mcm001/photonvision/tree/2023_10_15-gstreamer