OpenCV - Enabling gstreamer

[Sn3akyP3t3]

Was there any major reason for disabling gstreamer and/or ffmpeg support from the OpenCV library?

As I understand it the gstreamer documentation for OpenCV is cryptic, but appears to be supported. This may help some teams with a need to stream to their dashboard, but not desiring to package things up within network tables.

I see these flags were the cause: -DWITH_GSTREAMER=OFF \

These packages will be necessary if going to enable gstreamer support: sudo apt-get -y install libgstreamer1.0-dev sudo apt-get -y install libgstreamer-plugins-base1.0-dev

[PeterJohnson]

I started with the same flags used for the RoboRIO OpenCV build, and the build infrastructure we started with made it difficult to add support for 3rd party libraries like ffmpeg and gstreamer. That being said, I recently revamped the build infrastructure and along with that enabled ffmpeg support, and it should be relatively easy to add gstreamer support as well.

I'm a little confused by what you mean by "need to stream to their dashboard but not desiring to package things up within networktables". Streaming doesn't require networktables, networktables just provides the discovery mechanism for the dashboard to find the stream.

[Sn3akyP3t3]

I haven't used network tables yet so perhaps I have a misunderstanding of them. I've only read that teams have struggled trying to deliver the image stream over to their dashboard and ended up using network tables to push that content over. I've also read that network tables have about 100ms of latency which probably makes for a bad lip sync video :) My comment was mainly directed at alleviating a need for network tables to deliver the video content if the streaming is done independently.

I'm curious why NI doesn't just embrace MQTT and JSON instead of insisting on continuation of using network tables.

[PeterJohnson]

The image stream is completely independent of NetworkTables; it's delivered as MJPEG over a dedicated HTTP connection.

NetworkTables is not used for image data at all; it simply provides a distributed hash table of keys and typed values (e.g. numbers, strings, etc). The only relation to video streaming is that NetworkTables provides the HTTP URLs for the cameras from the device hosting the stream to the dashboards so the dashboards know where to fetch the MJPEG stream from.

NetworkTables was created by WPI, not NI (NI just built the LabView implementation). At this point NetworkTables has its own ecosystem of applications, and has well understood behavior and interaction with the robot WiFi environment along with controlled latency (striking a balance between the two is not something most protocols provide, but NetworkTables does). Switching to a different protocol such as MQTT would require significant effort and experimentation, and it's unclear how it would provide a significant benefit to teams over continuing to use NetworkTables. Also, why specifically MQTT over ZeroMQ or several other alternatives? That's not to say that there's not a place for pub/sub, and it's something we've looked at a number of times, but we've not convinced ourselves we need it yet.

Regarding NetworkTables latency, the standard update rate is 100 ms, so the worst case latency is that, but explicit flush commands can be used following writes to lower latency to essentially zero.

[Sn3akyP3t3]

Now I understand what you meant by using NetworkTables to announce the URL of the stream. I didn't understand that before, but that makes sense. I've still read a handful of forum posts and conversations where teams were attempting to deliver image data via NetworkTables so that didn't help since I don't have experience with them.

Perhaps this cartoon has some meaning to the situation of selecting an existing protocol: https://deepstreamhub.com/blog/an-overview-of-realtime-protocols/

My recommendation for MQTT stems from my experience to date with home automation. It seems to crop up a lot in IOT conversations and tools. I believe in the idea of pave the cowpaths which seems to be how NetworkTables came to be. I didn't realize other protocols were analyzed and not found to be suitable. Newer protocols appear to be showing up that may be more suitable for a second review. For example, I didn't know LWM2M was a thing.