Cannot capture RAW10 data using v4l2-ctl

[dawhite]

Getting this sensor up and running on the Jetson Nano Dev Kit was a painful process, which included having to completely rebuild JetPack 4.4 using the "Option B" instructions since the "Option A" Debian packages are out of date. If you try using them, they downgrade L4T and brick the Nano--you don't get past the first NVIDIA splash screen.

I finally did get the sensor working with the usual "video/x-raw" GStreamer pipelines using nvarguscamerasrc and the NV12 output format, but when I switched to "video/x-bayer" with v4l2src and the rggb output format, the pipeline would instantly fail. I did some research and found your fix for that, where you have to rebuild GStreamer so it can handle RAW10. I implemented that fix and instead of failing, the pipeline just hangs. So I went back to basics and tried capturing the raw data stream using v4l2-ctl directly instead of relying on GStreamer, to make sure that there wasn't an underlying problem. It also hangs, so I think the problem is with more fundamental v4l2 support. Here's what I'm seeing:

$v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.bayer --verbose VIDIOC_QUERYCAP: ok VIDIOC_S_EXT_CTRLS: ok VIDIOC_G_FMT: ok VIDIOC_S_FMT: ok Format Video Capture: Width/Height : 4032/3040 Pixel Format : 'RG10' Field : None Bytes per Line : 8064 Size Image : 24514560 Colorspace : sRGB Transfer Function : Default (maps to sRGB) YCbCr/HSV Encoding: Default (maps to ITU-R 601) Quantization : Default (maps to Full Range) Flags : VIDIOC_REQBUFS: ok VIDIOC_QUERYBUF: ok VIDIOC_QBUF: ok VIDIOC_QUERYBUF: ok VIDIOC_QBUF: ok VIDIOC_QUERYBUF: ok VIDIOC_QBUF: ok VIDIOC_QUERYBUF: ok VIDIOC_QBUF: ok VIDIOC_STREAMON: ok

Nothing happens after that last line--it just sits there. I have verified that v4l2-ctl can recognize the sensor as a source:

$v4l2-ctl -d /dev/video0 --list-formats-ext ioctl: VIDIOC_ENUM_FMT Index : 0 Type : Video Capture Pixel Format: 'RG10' Name : 10-bit Bayer RGRG/GBGB Size: Discrete 4032x3040 Interval: Discrete 0.033s (30.000 fps) Size: Discrete 1920x1080 Interval: Discrete 0.017s (60.000 fps)

I purchased the IMX477 sensor because I need to capture 10-bit 4K video and it looked like your driver made that possible, but right now I'm stuck with 8-bit 4K video. Your assistance is greatly appreciated.

[SLyra21]

Hi dawhite,

funny, I have the exact same problem, but I could not find any solution so far!

Best,

[rrcarlosrodriguez]

Hi @dawhite and @SLyra21 , sorry for the slow response. Could you please try the following?

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040 --set-ctrl bypass_mode=0 --stream-mmap

For some reason, specifying the format is not working properly with v4l2-ctl, but just get rid of it in the command.

[SLyra21]

Hi @rrcarlosrodriguez,

thank you for your help! I was able to write a raw image and display it afterwards. Now I would like to get a live stream, which can be displayed direct, e.g. with opencv etc. . I know, it will be 8bit then, but I need the raw image for processing! Do you have any idea how to do that? I tried to build a gst pipeline, but I was not succesful so far.

Best,

[rrcarlosrodriguez]

Hi @SLyra21 ,

You can try using v4l2src with GStreamer, but you will need to patch the element. Here are some instructions to do so:

https://developer.ridgerun.com/wiki/index.php?title=Compile_gstreamer_on_Jetson_TX1_and_TX2#Steps_to_patch_GStreamer_to_support_RAW10

[dawhite]

Appreciate the response. Still hangs for me at the same point, can't write a raw image. I tried your exact suggestion and this modified version of my original command, where I removed the pixel format:

$v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=1 --stream-to=test.bayer --verbose

The destination file is opened, but nothing is written. It just sits there until I kill the process with CTRL-C.

I'm wondering if the issue is related to what is discussed in this NVIDIA Developer Forums thread:

https://forums.developer.nvidia.com/t/why-use-v4l2-ctl-command-get-raw-data-is-alway-zero-at-jetson-tx1-r28-2/60345

Also, to use this camera module with the Jetson Nano, I'm stuck on JetPack 4.4.0 and have to build everything myself. I noticed that there is now official support for the Nano 2GB:

https://developer.nvidia.com/embedded/downloads#?search=imx477

When will there be official support for the regular Nano?

[rrcarlosrodriguez]

Hi @dawhite I gave a try to your command in the latest Jetpack 4.4.1 update and it works fine. You can find the updated patches in the repo.

[dawhite]

Thanks for checking! I was able to get the IMX477 to run under JetPack 4.4.1 using the updated patches, which is really helpful. I have not tried the command yet, but will do so. I also noticed that JetPack 4.5 was released a few days ago. Do you have plans to release patches for that and/or have NVIDIA incorporate support into their Nano baseline (4GB, not just 2GB) so you don't have to keep releasing patches?

[rrcarlosrodriguez]

@dawhite , we will keep releasing patches until NVIDIA incorporate the changes in their baseline. We don't have an ETA for that

[dawhite]

Tried the command again running under your patch for JetPack 4.5:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=4032,height=3040 --set-ctrl bypass_mode=0 --stream-mmap --verbose

It's still hanging in the exact same place (right after "VIDIOC_STREAMON: ok" prints to the terminal). I am unable to capture even a single frame.

[dustinkerstein]

@dawhite were you ever able to get this working? I can't seem to get bayer output on my Arducam IMX477 + Nano.

[dawhite]

@dustinkerstein no, I never got this to work. I kept things as simple as possible, even going so far as starting over with a new system image and installing the patch to enable support for the sensor, to make sure that there were no other possible sources of error. Still nothing. Just hangs....

[dustinkerstein]

@dawhite It's possible it's related to this issue - https://forum.arducam.com/t/imx477-driver-not-installing-correctly-on-r32-6-1/1681/21 - Have you tried with the latest Nvidia provided driver (which is limited to 3840x2160)?

[dawhite]

I haven't tried that yet. My entire setup is built around using the full resolution of the sensor. Still no response from the owners of this repo.

[dustinkerstein]

Check the latest driver posted in that above thread. Arducam got the full resolution raw bayer capture working at 15fps with v4l2-ctl.