Status of support for non-FFMPEG OpenCV backends?

[musicinmybrain]

I noticed “Enforced FFMPEG backend.” in the release notes for 6.2.2.

In Fedora Linux, we historically don’t have an ffmpeg package due to patent issues. Some users install it from RPM Fusion, but packages in the base distribution can’t rely on it. A stripped-down version was recently added to the distribution as ffmpeg-free, but there are no plans to try to build OpenCV with it.

Thus, Fedora fasttrack package users will typically be using the GSTREAMER backend.

---

With 6.2.2 and 6.2.3, I find that, in the headless build environment, I encounter three test failures:

[  FAILED  ] VideoReaderTest.UMatVideo
[  FAILED  ] VideoReaderTest.MatVideo
[  FAILED  ] VideoReaderTest.CountVideo

VideoReaderTest.log

With a “local” build on my workstation with hardware direct rendering support, I get two test failures:

[  FAILED  ] VideoReaderTest.UMatVideo
[  FAILED  ] VideoReaderTest.MatVideo

and the details are different: VideoReaderTest-local.log

All of these tests are new in 6.2.2, of course.

If I ignore or skip failing tests, and try opening the included test.avi with the resulting FastTrack build, it seems like basic functionality works. (There are probably some things I don’t know enough to look for.)

---

I guess what I’m trying to understand before putting more effort into understanding, skipping, and/or fixing these tests is whether ffmpeg support in OpenCV is intended be seen as a hard requirement for FastTrack going forward? If so, that’s very understandable, but would probably mean the base Fedora Linux distribution is not the right place for FastTrack.

[bgallois]

FFmpeg is enforced if found in the system. Otherwise, OpenCV will choose an available backend and the behavior is exactly the same as before. This was introduced to have consistent performance between Windows, macOS and Linux versions. Using GStreamer will normally work to open videos. On my system, GStreamer cannot open image sequences, but it seems that on Fedora the test is passing…

For what I can see, test are failing due to "GStreamer-CRITICAL" error https://github.com/opencv/opencv/issues/10324, but still OpenCV is able to open the video ignoring this error on my system, but the video cannot be browsed using the UI. I will perform some tests on a system with no FFmpeg to see if videos and image sequences can be opened correctly.

[bgallois]

After some testing on a vanilla Fedora + dnf install gstreamer*:

• Image sequences can be opened with the CAP_IMAGE backend (default behavior when FFmpeg is not found).
• Videos cannot be opened using CAP_GSTREAMER, due to error your GStreamer installation is missing a required plugin and by any other backend found on Fedora.

For what I see, there are several problems related to GStreamer because on a vanilla Fedora I can't open the test.avi at all. On my workstation (Arch Linux), the video open, but it is impossible to select a specific frame with cv::CAP_PROP_POS_FRAMES (GStreamer-CRITICAL **: gst_query_set_position: assertion 'format == g_value_get_enum (gst_structure_id_get_value (s, GST_QUARK (FORMAT)))' failed) function that the FastTrack UI uses a lot.

Hence, I think that with in the current state of the cv::VideoCapture with the GStreamer, it is difficult to guaranty that FastTrack without Ffmpeg will work with videos as intended.

[musicinmybrain]

Thanks for investigating! I think that’s pretty conclusive that ffmpeg is pretty much a hard dependency. It seems to make a difference whether an ffmpeg package is installed, even if OpenCV isn’t built with support for using it directly as a backend, and unfortunately the new ffmpeg-free doesn’t seem to suffice.