Now there is no way to control GStreamer drop timeout when ip-camera drops

[qwertyfive]

Descripe the feature and motivation

I'm using OpenCV with GStreamer, and sometimes my camera can disconnect. And when its happened, i just have program freezing for 20 s. And (maybe i didn't find) there is no way to change that timeout. Similar thing - is CAP_PROP_READ_TIMEOUT_MSEC for ffmpeg, but it didnt work for GStreamer.

Additional context

No response

[kallaballa]

Hi! I am a contributor. The obvious thing to do would be to set a timeout on the v4l2src element, but it simply doesn't provide that option (unlike e.g. udpsrc). Maybe you could workaround it using the watchdog element

for example:

cv::VideoCapture capture("v4l2src device=/dev/video1 ! watchdog timeout=1000 ! appsink", cv::CAP_GSTREAMER, {});

[qwertyfive]

Actually, im using rtspsrc and there is some timeouts, but they do not change anything(like tcp-timeout which is 20s just reduce open timeout, but not read)

[kallaballa]

And did you try watchdog as an alternative?

[qwertyfive]

Hey, it actually worked! Thank you so much, i`ve been looking for solution for 2 days! Thank you so much!

[qwertyfive]

Hmm, it worked with cmd directly to gstreamer, but still have issue with opencv. That strange!

[qwertyfive]

My pipeline:

string ns = " rtspsrc protocols=tcp location=" + s + " retry=1 latency=0 tcp-timeout=1 buffer-mode=1 ! watchdog timeout=1000 ! decodebin ! videoconvert ! videorate ! video/x-raw,framerate=25/1 ! appsink";

[qwertyfive]

Interesting thing - what happened when we write incorrect pipeline? Why and how opencv ignore wrong parts? Maybe this is the key? Like when i try watchdog timeout=1000 ! rtspsrc location=rtsp://my-link:8554 latency=0 ! decodebin ! autovideosink (console), it drops me with error (Could not link watchdog to rtspsrc), but when i try this with opencv, it just timeout after watchdog timeout(after begining of program)

[kallaballa]

Could you please:

• Post a full (but small) code example that exhibits the behavior
• Enable debug output by exporting the following variables before running your program:

  export OPENCV_LOG_LEVEL=VERBOSE
  export OPENCV_VIDEOIO_DEBUG=1
  export OPENCV_VIDEOCAPTURE_DEBUG=1

• Post the program output with debug output enabled

[qwertyfive]

Im suppose to do this on CMake list, am I? Cause im using Windows OpenCV build, then add libs and include to my project with VS 19, and dont understand, how am i suppose to do second part of your ask)

[kallaballa]

Oh, I didn't catch that you are on Windows. Instead of export you can use the cmd command set or the GUI

I can't test it on my side, but the following should work:

set OPENCV_LOG_LEVEL=VERBOSE
set OPENCV_VIDEOIO_DEBUG=1
set OPENCV_VIDEOCAPTURE_DEBUG=1
your_command.exe

[qwertyfive]

Log.txt My code:

#include <iostream>
#include <opencv2/opencv.hpp>

using namespace std;
using namespace cv;

int main()
{
    string s = "rtsp://192.168.144.25:8554";
    string ns = " rtspsrc protocols=tcp location=" + s + " retry=1 latency=0 tcp-timeout=100000 buffer-mode=1 ! watchdog timeout=2000 ! decodebin ! videoconvert ! videorate ! video/x-raw,framerate=25/1 ! appsink";

    VideoCapture cap(ns, CAP_GSTREAMER);
    Mat frame;

    if (!cap.isOpened())
    {
        cout << "NO CONNECTION" << endl;
        return -1;
    }

    while (cap.isOpened())
    {
        cap >> frame;
        if (frame.empty())
        {
            cout << "EMPTY FRAME" << endl;
            return -1;
        }

        imshow("GG", frame);
        waitKey(1);
    }
    return 0;
}

[kallaballa]

The pipeline doesn't seem to be the problem, because the following works for me (on Linux):

cv::VideoCapture capture("rtspsrc protocols=tcp location=rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 retry=1 latency=0 tcp-timeout=100000 buffer-mode=1 ! watchdog timeout=2000 ! decodebin ! videoconvert ! videorate ! video/x-raw,framerate=25/1 ! appsink", cv::CAP_GSTREAMER, {});

Could you try again and use the rtsp-url i used? rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4

[qwertyfive]

Same problem. Did u disconnect your network when testing this? I have this issue when i disconnect my camera(or when i trying test rtsp stream and then just drop my internet connection)

[kallaballa]

Yes I did, and watchdog worked.

[qwertyfive]

I`ve tried that on my friend linux pc, and he still have same issue

[kallaballa]

I don't fully understand what your problem at the moment is. The log file you supplied looks good. Stream is running until watchdog triggers and ends it.

[qwertyfive]

u can see, that there is no logs between 6 and 26 sec - this is when programm freeze

[qwertyfive]

What happened, if you play this test stream and then enable airplane mode?

[kallaballa]

You are right. I have the same problem now. Investigating

[kallaballa]

There seems to be a problem with error propagation because this works (it terminates ~2 seconds after i shut down my network interface):

gst-launch-1.0 -v rtspsrc protocols=tcp location=rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 udp-reconnect=false udp-buffer-size=1024 timeout=0 retry=0 latency=1000 do-retransmission=false buffer-mode=0 ! watchdog timeout=2000 ! decodebin ! videoconvert ! videorate ! video/x-raw,framerate=25/1 ! autovideosink

While this doesn't (watchdog triggers but no immediate reaction to the error on the gstreamer-bus):

cv::VideoCapture capture("rtspsrc protocols=tcp location=rtsp://wowzaec2demo.streamlock.net/vod/mp4:BigBuckBunny_115k.mp4 udp-reconnect=false udp-buffer-size=1024 timeout=0 retry=0 latency=1000 do-retransmission=false buffer-mode=0 ! watchdog timeout=2000 ! decodebin ! videoconvert ! videorate ! video/x-raw,framerate=25/1 ! appsink", cv::CAP_GSTREAMER, {});

[qwertyfive]

Yeah, this is why i`ve been so happy yesterday, when u told me about watchdog. It works directly with gstreamer(from cmd or terminal), but have some troubles with opencv(

[kallaballa]

I am familiar with Gstreamer but not with the OpenCV/Gstreamer backend. I need time... wanted to read that code anyway. Maybe in the meantime someone more enlightened can help :)

[qwertyfive]

I hope so) Maybe someone else notice that issue)

[asmorkalov]

Some findings on the issue: According to GStreamer documentation, watchdog sends message to the pipeline bus, but does not change pipeline state. Proof: https://gstreamer.freedesktop.org/documentation/debugutilsbad/watchdog.html?gi-language=c. OpenCV does not have permanent bus event handler, but check messages time-to-time in open/close to check operation status. See https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_gstreamer.cpp#L2376 and how it's used. OpenCV grabFrame waits for the next frame with gst_app_sink_pull_sample which does not have timeout and fails only if the pipeline changes its state. See https://github.com/opencv/opencv/blob/4.x/modules/videoio/src/cap_gstreamer.cpp#L494.

Options I see:

• replace gst_app_sink_pull_sample with pull-sample signal with timeout: https://gstreamer.freedesktop.org/documentation/applib/gstappsink.html?gi-language=c#GstAppSink::pull-sample.
• Hack the issue from pipeline side. GStreamer rtsp client options, global env, etc

[qwertyfive]

Hmm, why they dont use gst_app_sink_try_pull_sample with some kind of timeout that can be set? I think that could help, i'll try this myself, and if this will help, it's gonna be good update for OpenCV?)

[qwertyfive]

Yeah, it works. Maybe we need request for that like - Change gst_app_sink_pull_sample for gstreamer backend to gst_app_sink_try_pull_sample with new addition Parameter like CAP_PROP_GST_READ_TIMEOUT?

[kallaballa]

I'm just a contributor and not familiar yet with the Gstreamer backend, but from what I've seen by now that sounds like a good idea.

[asmorkalov]

Sounds like good solution. I'll take a look on technical details and return back with patch. Fill free to contribute, if you are ready.

[qwertyfive]

No, thank you, i'm not sure i can do that properly)

[asmorkalov]

@qwertyfive I added timeouts in https://github.com/opencv/opencv/pull/22919. Could you try my branch with timeouts and report status to the PR.

[asmorkalov]

Merged

[asmorkalov]

BTW, You need GStreamer 1.10 or newer to use the feature.

[linglongxian]

@qwertyfive hello, did you solve the problem? if yes, could you share your solution?

[qwertyfive]

@qwertyfive hello, did you solve the problem? if yes, could you share your solution?

Already added in #22919, so you can use it

[qwertyfive]

@qwertyfive hello, did you solve the problem? if yes, could you share your solution?

use CAP_PROP_OPEN_TIMEOUT_MSEC and CAP_PROP_READ_TIMEOUT_MSEC

[linglongxian]

@qwertyfive ok, thanks for your reply. i will rebuild my opencv and test it