Performance Drop after Second Pipeline Start

[robbyfrt]

Hi Allied Vision DevTeam,

I am unsure whether this is truly related to gst-vimbasrc, but I am experiencing a drop in FPS when I access gst-vimbasrc consecutively via OpenCV. See below a MWE of my OpenCV pipeline.

The first exectution returns 13.7 FPS (72ms), second execution reduces FPS to 5.2FPS (192ms). I can reach 13 FPS again when replugging the USB3 Camera (Alvium 1800U-500c).

Questions:

• Is cap.release() enough to gracefully disconnect from the camera? I suspect that some connection is not closed properly.
• I have no easy way of checking this in a gstreamer-only pipeline as I cannot get fpsdisplaysink to run in this pipeline.
• Do you have suggestions regarding my capture pipeline gst_str? Can I speedup my pipeline using a different videoformat?
• I work on a Jetson Xavier so I tried `nvvidconv ! 'video/x-raw(memory:NVMM) ! appsink' to pass images allocated in GPU memory directly to OpenCV without success. This is probably a OpenCV limitation.

import cv2
settings_file="/home/nvidia/optical_qc/jetson/acquisition/camera_acquisition_config.xml"
gst_str = f"vimbasrc camera=DEV_1AB22C00BD8C settingsfile={settings_file} ! video/x-raw,format=RGB ! videoconvert ! queue ! appsink"
cap = cv2.VideoCapture(gst_str, cv2.CAP_GSTREAMER)
tm = cv2.TickMeter()
count = 0
if cap.isOpened():
    while True:
        tm.start()
        ret_val, img = cap.read()

        cv2.waitKey(10)

        print(f"frame: {count}, shape: {img.shape}, fps {tm.getFPS():.2f}, execution_time {(tm.getAvgTimeMilli()):.0f}ms")
        count += 1

        if count == 100:
            break
        tm.stop()      
else:
    print("rtsp open failed")

cv2.destroyAllWindows()
cap.release()

[NiklasKroeger-AlliedVision]

Is cap.release() enough to gracefully disconnect from the camera? I suspect that some connection is not closed properly.

Unfortunately I do not have any experience using GStreamer pipelines as OpenCV capture devices. So I do not know if cap.release is enough. Perhaps turning on logging would help to see what exactly is going on. When the camera connection is shut down, you should see a INFO message that reads something along the lines of "Closed camera DEV_1AB22C00BD8C".

Turning on debug output should be as simple as setting the environment variable

GST_DEBUG=vimbasrc:DEBUG

If you want to go even further with your output (e.g. get a line printed for every frame that is received from the camera, you can also set the variable to GST_DEBUG=vimbasrc:DEBUG. But the debug setting above should be enough to get some more insights into what is happening, so try that one first.

Since you mention that you need to reset the camera by unplugging it, maybe it is one of the camera features you set via your settingsfile. Have you tried loading the settingsfile with the Vimba Viewer and letting frame acquisition run there to see what the fps is? Maybe some kind of auto exposure limits the speed at which frames can be recorded....

I have no easy way of checking this in a gstreamer-only pipeline as I cannot get fpsdisplaysink to run in this pipeline.

13fps and 5fps seems like a pretty drastic difference that should be visible to the naked eye. Do you see a difference in performance just by looking at it if you simply use a regular xvimagesink to display your recorded frames?

Do you have suggestions regarding my capture pipeline gst_str? Can I speedup my pipeline using a different videoformat?

The pipeline itself seems fine to me. The camera you are referencing (1800 U-500c) supports a number of yuv formats. I am not sure if there are maybe some more optimized algorithms at play if you try to transfer data as one of those? Especially the sub-sampled ones might lead to higher performance as less image data needs to be transmitted compared to full RGB frames. This however does obviously come at the cost of reducing spectral resolution in your recorded image. Depending on your application this might be acceptable or not. This is up to you. Maybe try the following GStreamer formats (for corresponding Vimba formats see table in README):

• IYU2
• UYVY
• IYU1

I work on a Jetson Xavier so I tried nvvidconv ! 'video/x-raw(memory:NVMM) ! appsink to pass images allocated in GPU memory directly to OpenCV without success. This is probably a OpenCV limitation.

As mentioned above I have unfortunately no experience using a GStreamer pipeline as OpenCV capture device. So I am afraid I cannot offer any help on this topic.

[robbyfrt]

Alright I found the mistake in the config file.

AcquisitionFrameRate was set to 5FPS...

The first time I run the script, this setting seems to be ignored and the acquisiotion is set to 13FPS. The scond time, the FrameRate is set with the rest of the parameters. This is still a bit unexpected but I resorted to a new config file and the problem was fixed. I can upload the config in case you are interested.

Thank you for the other tips.

• GST_DEBUG suggests that everything is closed as it should after my OpenCV pipeline.
• It seems that Gstreamer is unable to pull data from NVMM storage. You do need to pass images located on memory using appsink. There is not much else you can do with the Gstreamer backend for Videocapture in OpenCV (see OpenCV Documentation)

I am closing this now as fixed, thank you!