What is the zedsrc gstreamer returned color format?

[Tetsujinfr]

hi

does the plugin return rgb or bgr color format? asking because I have sometimes so strange artefacts, which are usually fixed by videoconvert but not always. In some deep learning models I have different behaviours when using v4l2src rather than zedsrc, at the same resolution, and I am not sure why. thanks

[Myzhar]

The ZED SDK provides frames in BGRA format and the zedsrc element simply copies the data in the same format. The format is declared in the element caps, so GStreamer can correctly handle the conversions: https://github.com/stereolabs/zed-gstreamer/blob/master/gst-zed-src/gstzedsrc.cpp#L513-L612 Data copy is performed here: https://github.com/stereolabs/zed-gstreamer/blob/master/gst-zed-src/gstzedsrc.cpp#L1956-L2035

[Tetsujinfr]

thanks I still struggle to get my app working with zedsrc while it works with v4l2src, that is running me nuts, I cannot see where the difference would come from.

Here are the 2 gstreamer commands I test. When I save the frame received by opencv in the app and save it down as a bmp I cannot spot fundamental differences in color format, but there is a difference in fov and in exposure (not sure why). It seems that zedsrc provides a perspective adjusted image while v4l2 does not.

-> Is there a way to de-activate the perspective correction of zedsrc?

test zedsrc: gst-launch-1.0 zedsrc stream-type=2 resolution=2 framerate=60 depth-mode=0 enable-positional-tracking=FALSE ! videocrop top=0 left=0 right=0 bottom=720 ! videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! xvimagesink test2 v4l2: gst-launch-1.0 v4l2src device=/dev/video1 ! videoscale ! video/x-raw, width=1280, height=720 ! videoscale ! video/x-raw,width=640,height=480 ! videoconvert ! xvimagesink

Images below (wider fov is with v4l2src):

[Myzhar]

@Tetsujinfr what you see is not a perspective correction, but a camera undistortion. The ZED SDK provides RGB rectified images using the camera pinhole model: https://docs.opencv.org/4.5.2/d9/d0c/group__calib3d.html If you look at the top edge of the box you can notice that v4l2src provides an image where it is slightly curved, while the rectified ZED SDK image provided by zedsrc is not affected by this distortion.

If you use also depth information this guarantees that the pixel of coordinates (u,v) of every left image matches the same pixel (u,v) of the depth map. If you are not using depth information and you want the raw distorted images (the same provided by v4l2src) you can replace every instance of sl::VIEW::LEFT and sl::VIEW::RIGHT with sl::VIEW::LEFT_UNRECTIFIED and sl::VIEW::RIGHT_UNRECTIFIED: https://github.com/stereolabs/zed-gstreamer/blob/master/gst-zed-src/gstzedsrc.cpp#L1956-L1990 then you must recompile and install the packages.

[Tetsujinfr]

Thanks, I am going to try that even if I suspect this is not the root cause of my issue. But I am trying to rule out all differences. It might be useful to add a gstr property in the future to enable (default) or not the distortion correction. I was really surprised to see v4l2 pull distorted frames, did not realize the camera was exhibiting those.

Do you know what pixel format v4l2src pull out of the camera? I have tried to check this out but unsuccefully (I may need to save raw opencv matrices and compare files but if you know that would be great).

[Myzhar]

v4l2src retrieves the image information in YUYV format that is the native image format provided by the UVC interface of the ZED cameras. You can get more information from the source code of the zed-open-capture repository: https://github.com/stereolabs/zed-open-capture/blob/master/src/videocapture.cpp#L425-L456

The Open Capture examples show how the YUYV image buffer is converted to BGR to be displayed on screen: https://github.com/stereolabs/zed-open-capture/blob/master/examples/zed_oc_video_example.cpp#L92

[Tetsujinfr]

thanks, that was helpful. So I am still debugging and I suspect my issue is simply linked to the sl::Mat vs cv::Mat format differences, which I too quickly overlooked. I found your article on how to deal with ocv with the sdk here

Now, I want opencv to deal with the frame capture of the zed through the zedsrc gstreamer interface, instead of using the grab function from the sdk. I have tried to use slMat2cvMat and apply it to the ocv captured frame (with zedsrc gstreamer), but unsuccessfully.

So, when I do:

cap = new cv::VideoCapture( GStreamer_string, cv::CAP_GSTREAMER);
cap.read(frame)

with GStreamer_string = 'zedsrc stream-type=0 resolution=3 framerate=100 depth-mode=0 enable-positional-tracking=FALSE ! videobox autocrop=true ! videoscale method=0 envelope=0 sharpness=0 ! video/x-raw, width=640, height=480 ! videoconvert ! appsink'

I assume that frame contains a sl::Mat frame( (640, 480) , MAT_TYPE::U8_C4) instead of a cv::Mat frame( (640, 480), CV_8UC1) (for YUV ) or cv::Mat frame( (640, 480), CV_8UC3) (for BGR) which I usually get through v4l2?

=> If correct, how can I convert properly this received zed frame through gstreamer into a standard ocv BGR matrix?

I have tried to leverage slMat2cvMat() but unsuccessfully so far.

thanks for your help