How to use HDMI2CSI for Tegra TX2 within OpenCV?

[zhangmGoL]

Hi, I have a problem when using the HDMI2CSI module within OpenCV. I test it in the following platform: (1) Platform: TX2 (2) OpenCV: 3.1 with cap_gstreamer support (gst version: 1.8.3) (3) HDMI2CSI Driver: Image_28-1_hdmi2csi (4) HDMI Input: the HDMI output of another PC Before using OpenCV for testing, the commands used in https://github.com/InES-HPMM/linux-l4t-4.4/wiki/hdmi2csi#examples (using gst-launch-1.0) are test, and I found that these commands work very well. Then I use OpenCV to capture the gst-pipeline-stream. The code is very easy and shown below: ################################################################################ static const string PIPELINE_NAME = "v4l2src device=/dev/video0 ! video/x-raw, width=1920, height=1280, framerate=60/1, format=UYVY ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! appsink sync=false";

class HdmiInputStreamer { public: static boost::shared_ptr Get() { static boost::sharedptr ptr; if (! ptr) { ptr.reset(new HdmiInputStreamer()); } return ptr; } // output frame void pop(cv::Mat* image) { cap >> *image; } // show void show() { while (1) { cv::Mat image; pop(&image); cv::namedWindow("HdmiInputStreamer", WINDOWAUTOSIZE); cv::imshow("HdmiInputStreamer", image); cv::waitKey(1); } } ~HdmiInputStreamer() { if (cap.isOpened()) { cap_.release(); } }

private: cv::VideoCapture cap; std::string pipeline; HdmiInputStreamer() { pipeline_ = PIPELINENAME;
while(! cap.open(pipeline_)) { LOG(INFO) << "The hdmi input stream is unreachable, please check and wait ..."; // sleep sleep(1000); } } }; ################################################################################
I use HdmiInputStreamer()::Get()->show() to display the input HDMI Stream, and the ERROR is shown below: GStreamer Plugin: Embedded video playback halted; module v4l2src0 reported: Internal data flow error. OpenCV Error: Unspecified error (GStreamer: unable to start pipeline ) in cvCaptureFromCAM_GStreamer, file /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp, line 818 terminate called after throwing an instance of 'cv::Exception' what(): /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp:818: error: (-2) GStreamer: unable to start pipeline in function cvCaptureFromCAM_GStreamer

Aborted (core dumped)

I have also tested some other pipelines with different names: (1) "v4l2src device=/dev/video0 ! video/x-raw, width=1920, height=1280, framerate=60/1, format=UYVY ! nvvidconv ! video/x-raw(memory:NVMM), format=BGRx ! videoconvert ! appsink sync=false" (video.bin:3726): GStreamer-CRITICAL **: gst_mini_object_copy: assertion 'mini_object != NULL' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_caps_get_structure: assertion 'GST_IS_CAPS (caps)' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_structure_copy: assertion 'structure != NULL' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_caps_append_structure_full: assertion 'GST_IS_CAPS (caps)' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_mini_object_unref: assertion 'mini_object != NULL' failed

(video.bin:3726): GStreamer-CRITICAL **: gst_mini_object_ref: assertion 'mini_object != NULL' failed GStreamer Plugin: Embedded video playback halted; module v4l2src0 reported: Internal data flow error. OpenCV Error: Unspecified error (GStreamer: unable to start pipeline ) in cvCaptureFromCAM_GStreamer, file /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp, line 818 terminate called after throwing an instance of 'cv::Exception' what(): /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp:818: error: (-2) GStreamer: unable to start pipeline in function cvCaptureFromCAM_GStreamer

Aborted (core dumped)

(2) "v4l2src device=/dev/video0 ! video/x-raw, width=1920, height=1280, framerate=60/1, format=I420 ! nvvidconv ! video/x-raw(memory:NVMM), format=BGRx ! videoconvert ! appsink sync=false"

The same error as (1)

(3) "v4l2src device=/dev/video0 ! video/x-raw, width=1920, height=1280, framerate=60/1, format=I420 ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! appsink sync=false" ./build/tools/video.bin GStreamer Plugin: Embedded video playback halted; module v4l2src0 reported: Internal data flow error. OpenCV Error: Unspecified error (GStreamer: unable to start pipeline ) in cvCaptureFromCAM_GStreamer, file /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp, line 818 terminate called after throwing an instance of 'cv::Exception' what(): /home/nvidia/Downloads/opencv_liliang/opencv/modules/videoio/src/cap_gstreamer.cpp:818: error: (-2) GStreamer: unable to start pipeline in function cvCaptureFromCAM_GStreamer

Aborted (core dumped)

The same error as in main test situation

(4) "v4l2src device=/dev/video0 ! video/x-raw, width=1920, height=1280, framerate=60/1, format=UYVY ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink sync=false"

The same error as in main test situation

I am not sure what the problem is, and any help are appreciated!

[tkammacher]

Hi The problem in your GStreamer pipeline is in the usage of the memory:NVMM attribute. Although it is not really documented by Nvidia, we have found that nvvidconv can only be used when either the input or the output is in the NVMM memory. Therefore all your pipelines, where both the input and the output are in the normal memory (without NVMM) will fail. Another issue is that videoconvert can not access the NVMM memory. That is the reason for the rest of your pipelines failing.

You could use only videoconvert instead of nvvidconv, but this will result in bad performance. But the good news is, that there is a workaround for using the nvvidconv: Just use it twice, once to get into NVMM and once to get out of it. An example is below:

v4l2src device=/dev/video0 ! 'video/x-raw, width=1920, height=1080, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=BGRx' ! nvvidconv ! 'video/x-raw, format=BGRx' ! appsink

We have successfully tested this pipeline in GStreamer, but not in an OpenCV application. Does this solve your issue?

[zhangmGoL]

Hi, @tkammacher ,Thanks for your help so much. I have tested your pipeline in OpenCV, with the following two methods: (1) CPU mode: "v4l2src device=/dev/video0 ! video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY ! videoconvert ! video/x-raw, format=BGRx ! videoconvert ! appsink"; (2) GPU mode: "v4l2src device=/dev/video0 ! video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY ! nvvidconv ! video/x-raw(memory:NVMM), format=BGRx ! nvvidconv ! video/x-raw, format=BGRx ! videoconvert ! appsink"; I found that (1) and (2) both work well, but (1) suffers great latency and buffering compared with (2), due to low performance of CPU processing. I still have some other problems when using (2), as follows:

1. The hdmi input resolution is 3840x2160 by default, when connected to a Host-PC-HDMI-Output. how to configure it to 1080p or 720p?
2. A horizontal scan line is always displayed in the frame, how does it happen and how to fix it?
3. When I change to HDMI-Input B port (1080p by default), I found that all commands above fail, is there any other steps I should do before I use port B? Thanks a lot!

[tkammacher]

Ok, good to hear that you can get some data into your OpenCV application.

The hdmi input resolution is 3840x2160 by default, when connected to a Host-PC-HDMI-Output. how to configure it to 1080p or 720p?

The HDMI2CSI board acts like a HDMI sink (e.g. a Display). Therefore it provides a EDID which contains all the timings (resolution & framerates) that it supports. The HDMI source then negotiates one of those timings. Per default we have set 3840x2160p30, but there are also other timings defined. You can see a list of all the timings and how to change the EDID on our Wiki: https://github.com/InES-HPMM/linux-l4t-4.4/wiki/hdmi2csi#changing-the-edid But probably you don't need to do that. Just set your HDMI source to the resolution that you want (e.g. 1920x1080p60) and then change the GStreamer pipeline to the same parameters (e.g. video/x-raw, width=1920, height=1080, framerate=60/1, format=UYVY).

A horizontal scan line is always displayed in the frame, how does it happen and how to fix it?

I have not seen this behaviour before. Can you send a screenshot of this?

When I change to HDMI-Input B port (1080p by default), I found that all commands above fail, is there any other steps I should do before I use port B?

To use HDMI Input B, you have to change device=/dev/video0 to device=/dev/video1 as described in the Wiki: https://github.com/InES-HPMM/linux-l4t-4.4/wiki/hdmi2csi#examples

[zhangmGoL]

According to your tips, I modified the EDID file to configure the default resolution to 1080p50 and 720p60, and I find it really works. The horizontal scan line looks like this: I found that when I change the resolution from 4K -> 1080p50 -> 720p60, the appearance of vertical asynchrony (I called this as horizontal scan line) is progressively reduced. I doubt it is relavant to the FPS & clocks settings. I analyzed the settings of EDID files: Is there something requirements to fulfilled for these parameters, to completely eliminate the vertical asynchrony?

[tkammacher]

This looks like some kind of screen tearing. I have not seen this before when capturing HDMI sources. Also it's interesting that it gets less problematic if you reduce the resolution. This should not have to do with the EDID.

Can you try capturing with GStreamer directly to a HDMI output (no OpenCV) and see if the screen tearing issue is still visible? This will show if the problem is introduced by OpenCV or if it is present already before. For example:

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=3840, height=2160, framerate=30/1, format=I420' ! nvoverlaysink sync=false

Is it possible that the problem is caused by your HDMI source? If you connect the HDMI source to a HDMI screen directly (without the HDMI2CSI in the middle), do you see the problem still? Can you try capturing with the HDMI2CSI and with a different HDMI source? a camera or a media box or a different PC/Laptop.. This should tell you whether the problem is caused by the source or the TX2/HDMI2CSI. E.g.

[zhangmGoL]

I found that the input HDMI source has screen tearing as well, it's not introduced by GStreamer or OpenCV. I will test it using external camera, to check if it works good without any other problem. I found that the latency of the conversion & display is about 80-90ms, it is very close to the reference value you have given, which is about 70ms. By removing the time of display, I think it should be consistent with your reference values. My problem is, is there any other method I can make use of, to further reduce the latency of the data conversion for a delay-sensitive apps, as what we are facing?
Thanks a lot for your help recently!

[tkammacher]

You can do some things to try and reduce latency, but I don't think you will be able to achieve any drastic reductions, unless you really go deep into the details. What order of latency are you trying to achieve? For extremely low latency applications, we usually use different architectures, e.g. FPGA based.

There are several sources for latency:

• HDMI-to-CSI conversion in the TC358840 chip. Can't do much about that, but probably it does not add that much.
• Capturing the frame on the TX1 (If you display at 30FPS, one frame lasts 33ms)
• Writing from the TX1 Video Input to memory. Here you can use the VI DMA (see io-mode=3 below)
• Colorspace conversion (certain video renderers require a specific color space, e.g. if you use nvoverlaysink, you need I420 or NV12)
• Of course the rendering to the display takes time. Is this necessary in your application or are you just displaying for debug purposes?
• Speed up the way that frames are passed from the V4L2 video source into your OpenCV application

So what you can try:

• Set your HDMI source and your GStreamer pipeline to a higher framerate. E.g. use 1080p60 instead of 1080p30
• Do not care about timestamps, just display buffers as fast as possible: Add sync=false to your video sink at the end of the pipeline
• Use Userptr-mode for passing buffers (io-mode=3). An example pipeline: gst-launch-1.0 v4l2src io-mode=3 device=/dev/video0 do-timestamp=true ! 'video/x-raw, width=3840, height=2160, framerate=30/1, format=UYVY' ! xvimagesink sync=false
  • For this to work, you need to rebuild GStreamer, as described in https://github.com/InES-HPMM/linux-l4t-4.4/wiki/CustomGStreamer
• Do not use a GStreamer pipeline (and appsink) for capturing, but use the opencv methods to capture from V4L2 interface. I don't know whether this is faster, but it could be worth a try. Using VideoCapture from OpenCV, I think: https://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#videocapture

[zhangmGoL]

I have tried the VideoCapture class provided by OpenCV to open the device, but it has very large latency compared with the GStreamer method, which has accelerated by NV. I found a new problem, I use GoPro camera HDMI-output to connect to port-A, and I found that the pipeline worked well under PC-HDMI-Source encounters a mistake, as follows: ##################################################### ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Device 'dev/video0' cannot capture at 1920x1080 Additional debug info: gstv4l2object.c(3469): gst_v4l2_object_set_format_full (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Tried to capture at 1920x1080, but device returned size 16x32 ##################################################### I have do the following tests: (1) connect GoPro camera HDMI output to a Display Monitor, the display parameter is 1080p60, it works well; (2) I changed the driver default resolution from 4Kp30 -> 1080p60 -> 720p60, all Errors are the same except the device returned size: 1080p60 returns 16x32 and 4Kp30 returns 640x480. I directly use GStreamer Commands to display. (3) I also used it in OpenCV apps, it returns GStreamer errors. (4) I found that use OpenCV VideoCapture (use V4L2 API) works well, but it has a latency greater than 200ms for 720p resolution. (5) The PC-HDMI-Output works well under the same configuration, but GoPro HDMI output does not. What is the difference between these two sources, and how to fix it?

[tkammacher]

As far as I know GoPros do not dynamically set their resolution according to the EDID. You have to set the output resolution on the camera.

I don't know how exactly you have changed you EDID, so I can not compare it. With the following setup we can capture 1080p60 from a GoPro 4. Please try this:

• Set the GoPro to output 1080p60 (if you only have options like 25/50, you must set your GoPro from PAL to NTSC)
• Use our default EDID. Although 2160p30 is the default, it also has many other timings enabled (for example 1080p60).
• Launch a GStreamer pipeline that captures 1080p60: gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=1920, height=1080, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=60/1, format=I420' ! nvoverlaysink sync=false

Does this solve the problem?

And another note: We have had problems with the GoPro and certain HDMI-cables. Please make sure to try out different HDMI cables!

[zhangmGoL]

I met a problem now, the HDMI2CSI board does not work suddenly, I do not know what's going on ... I use PC-HDMI-Source, and it does not work yet. The Errors are shown below: Commands used: gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw, width=1920, height=1080, framerate=60/1, format=UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=1920, height=1080, framerate=60/1, format=I420' ! nvoverlaysink sync=false ###################################################################### Setting pipeline to PAUSED ... Pipeline is live and does not need PREROLL ... Setting pipeline to PLAYING ... New Clock: GstSystemClock [Now the program is blocked here ...] ###################################################################### The program is always blocked when showing "New Clock: GstSystemClock", what's going on during the block interval ...? Has the Conversion Module damaged? Can I do something to repair it?

[tkammacher]

Hi, please try using a configuration that previously worked (Driver, EDID, GStreamer Pipeline,..). If that does not work, try using different HDMI sources and HDMI cables to eliminate external Hardware issues.

If the issue persists, can you send the verbose output of the GStreamer pipeline and then send the kernel log. To get the verbose GStreamer output, add -vvv to the end of the pipeline. To show the last 100 lines of the kernel log: dmesg | tail -n 100

Hopefully these log files give some information about the problem.

[tkammacher]

No news after a month, therefore close this issue. If the problem still exists, re-open the issue..