Add support for libcamera

fangchenli commented 2 years ago

The new Raspberry Pi OS Bullseye comes with a new open camera stack libcamera. It would be nice if opencv could add support for it in VideoCapture. libcamera also lists integration with opencv as a project idea for Google Summer of Code 2022 on their website.

StefanBruens commented 2 years ago

The https://github.com/opencv/opencv/labels/platform%3A%20arm label is somewhat incorrect.

While libcamera is more or less a hard dependency for platform agnostic (i.e. not SOC dependent) support of MIPI CSI cameras, and often these are found on embedded ARM SBCs, there are other use cases as well:

libcamera also supports USB cameras (see e.g. https://libcamera.org/open-projects.html#adding-uvc-hardware-timestamp-support)
MIPI CSI is also found on RISC-V and x86 (embedded) platforms

kbingham commented 2 years ago

https://github.com/kbarni/LCCV looks like it has some development towards this - but I'm not sure why it's being done externally as a wrapper rather than adding support directly to openCV.

kbarni commented 2 years ago

I'm the developer of LCCV, and I agree that it would be best if the VideoCapture module would support libcamera as backend. I can help in solving this issue. And you are more than welcome to reuse my code.

ShahudullahKhan commented 1 year ago

I'm the developer of LCCV, and I agree that it would be best if the VideoCapture module would support libcamera as backend. I can help in solving this issue. And you are more than welcome to reuse my code.

@alalek @asmorkalov @StefanBruens Is there any update on this issue? Your early response will be highly appreciated. Thanks.

asmorkalov commented 1 year ago

No progress with this ticket for now. OpenCV is open source project. Feel free to contribute a solution.

kbingham commented 1 year ago

Note that I believe opencv can also use gstreamer sources, so using our libcamera gstlibcamerasrc element could already be possible.

There are some current known limitations in the gstreamer element, such as color space managment, and frame rate control - but those are being developed right now and should be getting integrated soon I hope.

I'd be happy to hear if anyone can test using the libcamera gstreamer element with opencv, and if you have any bugs or issues - report them here, or in our irc channel, or bugs.libcamera.org.

And if anyone would like to work towards native libcamera support in openCV - I'm happy to help there too.

krjw-eyev commented 1 year ago

@kbingham Do you know when those changes to gstreamer will be integrated? It was supposed to be implemented during the Google code of summer last year if I understood correctly.

kbingham commented 1 year ago

I thought they were already merged. What are you testing with ?

krjw-eyev commented 1 year ago

I am on a raspberry pi and I am interested in having 120FPS which the camera supports.

kbingham commented 1 year ago

that sounds like something you should take up with the raspberry pi folk on their camera forums.

https://forums.raspberrypi.com/viewforum.php?f=43

orowith2os commented 11 months ago

Adding support for PipeWire may be preferable, this blog post from someone in GNOME goes into more detail. To sum it up:

If you made this drawing of what the current state is, then thanks to our backwards compatibility with ALSA, PulseAudio and Jack, all the applications would be pointing at PipeWire for their audio handling like they are in the illustration you see above, but all the video handling from most applications would be pointing directly at v4l2 in this diagram. At the same time we don’t want applications to port to libcamera either as it doesn’t offer a lot of the flexibility than using PipeWire will, but instead what we propose is that all applications target PipeWire in combination with the video camera portal API.

I've already made the relevant issue: https://github.com/opencv/opencv/issues/23045

kbingham commented 11 months ago

OpenCV is probably one of the frameworks I'd think merits direct libcamera integration (as well as PipeWire integration), as it may be used in more embedded use cases potentially without a full desktop manager or PipeWire environment.

We have a Google Summer of Code project looking at this currently.

orowith2os commented 11 months ago

@kbingham sounds good, I'd love to keep an eye on it if there's anything publicly available. It may be useful to contact some PipeWire and libcamera folks about how to handle it too.

kbingham commented 11 months ago

We're always available for real-time chat in https://matrix.to/#/#_oftc_#libcamera:matrix.org and the pipewire guys are at https://matrix.to/#/#pipewire:matrix.org The GSoC OpenCV project has only just started a week ago so it will be a few weeks until there's something to report. I've asked that they reach out to the OpenCV project once there is some basic plumbing available.

sravich1 commented 8 months ago

Hi, I am currently using openCV where I am using cv2.videocapture() to get the video stream in Raspberry pi 4 /compute module. I am not able to add the /dev/video... of the raspberrypi bulls eye OS. so looks like openCV is not compatible with cv2 so what are the laternative solution so that we could use openCV in bulls eye.. any inputs will help us a lot. Thanks Ravi

peglegsqueeks commented 6 months ago

With the release of the Raspberry Pi 5 using Debian Bookworm and Python 3.11.2 Many people will be having an issue if they are using openCV. It would really help a LOT of people if openCV supported Libcamera natively.

ddbaron commented 5 months ago

OpenCV is probably one of the frameworks I'd think merits direct libcamera integration (as well as PipeWire integration), as it may be used in more embedded use cases potentially without a full desktop manager or PipeWire environment.

We have a Google Summer of Code project looking at this currently.

@kbingham Is this going to be integrated? Any idea when it will be available for use? ref: https://github.com/advait-0/opencv/tree/libcamera-final ref: https://advait-0.github.io/GSoC23/Final-Report/

kbingham commented 5 months ago

@advait-0 What is the status of your work? Did you ever post it to the OpenCV project?

advait-0 commented 5 months ago

@advait-0 What is the status of your work? Did you ever post it to the OpenCV project?

Hello, yes the integration does work but at the moment although only for MJPEG and YUYV and unfortunately for specific set resolutions. As discussed with Uajain I was planning to create a pull request after adding NV12 support and fixing the resolution issue(this being the bigger issue requiring some restructuring) .

Sorry I was caught up in something else but will post a fix by month's end. Regards

advait-0 commented 4 months ago

Quick update: Most of the issues mentioned are now resolved, I'm working on testing the NV12 support. I'm in touch with the libcamera team and hopefully we will have a working framework posted soon.

If there is anything specific that you wish to add support for please let me know and I'll try to add it if possible.

jhautbois commented 4 months ago

@advait-0 Hi there ! Any branch we could use somewhere to test it ? Even if NV12 is not supported yet, I am interested in testing it, and can help if needed ;-).

advait-0 commented 4 months ago

Yes @jhautbois you can checkout this branch https://github.com/advait-0/opencv/tree/libcamera-final While building with cmake use -DWITH_LIBCAMERA=ON. I'm working to find a fix for this but when using cap.read you need to check if(cap.read(frame). Here is a simple program you can you to test your build: https://paste.debian.net/1303027/

jhautbois commented 4 months ago

Well, @advait-0, I had a look and did a few tests, some work is still missing obviously ;-). First, we can't start/stop multiple times, there is no nice way to stop the capture and start it again, did you expect the user to always stop its application ? Or did I miss something ?

Next, it seems that YUYV is not working, as asking for it always returns MJPEG on my laptop. The cam application is able to do it though:

$ cam -c1 -s role=viewfinder,width=1280,height=720,pixelformat=YUYV -C
[8:18:02.819608801] [165669]  INFO Camera camera_manager.cpp:284 libcamera v0.0.0+4501-e8f20401
Camera configuration adjusted
Using camera \_SB_.PC00.XHCI.RHUB.HS06-6:1.0-0c45:6a14 as cam0
[8:18:02.840298592] [165669]  INFO Camera camera.cpp:1183 configuring streams: (0) 1280x720-YUYV

I understand it is work in progress, but I was expecting a bit much stable work :-/

advait-0 commented 4 months ago

Thanks for the output @jhautbois, i've committed a change so YUYV should be working on selection now. Yes I realize it still needs quite some work, sorry it's not too stable yet :/. I'm still learning, but I hope i'll be able to finish it all :). I haven't tested with multiple cameras yet, I intended to have a stable base first, but i'll do that soon.

Could you please send a sample use case where you try starting and stopping multiple times, so I can understand your intended usage better.

samuk commented 3 months ago

following

peglegsqueeks commented 3 months ago

Hi, just wanted to inquire has there been any progress on this?

Everyone who uses a Raspberry Pi 5 and a camera can't use Opencv with "VideoCapture(0)" at the moment as it returns an empty frame.

Thanks

swdee commented 3 months ago

@advait-0 I compiled the branch from https://github.com/advait-0/opencv/tree/libcamera-final and test code from https://paste.debian.net/1303027/ however upon executing it exits with a segmentation fault.

./example 
[6:18:20.156221133] [12518]  INFO Camera camera_manager.cpp:297 libcamera v0.0.5+83-bde9b04f
[6:18:20.195817636] [12519]  WARN RPI vc4.cpp:383 Mismatch between Unicam and CamHelper for embedded data usage!
[6:18:20.197058585] [12519]  INFO RPI vc4.cpp:437 Registered camera /base/soc/i2c0mux/i2c@1/imx219@10 to Unicam device /dev/media2 and ISP device /dev/media5
Cam init called
Entered setProperty
Entered icvSetFrameSize
icv Width set value: 643
Entered setProperty
Entered icvSetFrameSize
icv Width set value: 643
icv Height set value: 362
icvSetFrame Height:362  Width:643
Entered setProperty
Entered setProperty
Backend: libcamera

True
Validating config in grabFrame
VideoRecording
Config details:643
Config details:643
Validated viewfinder configuration is: 643x362-MJPEG
[6:18:20.198443644] [12518]  INFO Camera camera.cpp:1033 configuring streams: (0) 643x362-NV12
[6:18:20.199031342] [12519]  INFO RPI vc4.cpp:565 Sensor: /base/soc/i2c0mux/i2c@1/imx219@10 - Selected sensor format: 1920x1080-SBGGR10_1X10 - Selected unicam format: 1920x1080-pBAA
Entered open
Segmentation fault

advait-0 commented 2 months ago

@swdee can you please send me the output for cam -c1 -I (replace the number in -c1 with the actual camera you're using). It's configuring your stream for NV12 and NV12 support isn't working as of now. If your camera supports MJPEG or YUYV you can try using that. Ill add a detailed documentation on my github soon so hopefully it will be easier for everyone to try out.

swdee commented 2 months ago

@advait-0 I am not sure what package provides the cam command, however the output from libcamera-hello --list cameras is;

0 : imx219 [3280x2464] (/base/soc/i2c0mux/i2c@0/imx219@10)
    Modes: 'SRGGB10_CSI2P' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
                             1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
                             1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                             3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]
           'SRGGB8' : 640x480 [103.33 fps - (1000, 752)/1280x960 crop]
                      1640x1232 [41.85 fps - (0, 0)/3280x2464 crop]
                      1920x1080 [47.57 fps - (680, 692)/1920x1080 crop]
                      3280x2464 [21.19 fps - (0, 0)/3280x2464 crop]

This is an Arducam IMX219 based camera connected to a Raspberry Pi CM4.

I have also tried on an official Pi v2 camera with IMX219 sensor on a Raspberry Pi 5. I compiled your OpenCV branch against the latest libcamera version v0.2.0+46-075b54d5 but when running the sample program I get a different problem.

[0:39:29.076406344] [11724]  INFO Camera camera_manager.cpp:284 libcamera v0.2.0+46-075b54d5
[0:39:29.084318557] [11727]  INFO RPI pisp.cpp:662 libpisp version v1.0.4 6e3a53d137f4 14-02-2024 (14:00:12)
[0:39:29.097018309] [11727]  INFO RPI pisp.cpp:1121 Registered camera /base/axi/pcie@120000/rp1/i2c@88000/imx219@10 to CFE device /dev/media0 and ISP device /dev/media2 using PiSP variant BCM2712_C0
Cam init called
Entered setProperty
Entered icvSetFrameSize
icv Width set value: 640
Entered setProperty
Entered icvSetFrameSize
icv Width set value: 640
icv Height set value: 480
icvSetFrame Height:480  Width:640
Entered setProperty
Entered setProperty
Backend: libcamera

True
Validating config in grabFrame
VideoRecording
Config details:640
Config details:640
[0:39:29.097515128] [11724]  WARN V4L2 v4l2_pixelformat.cpp:346 Unsupported V4L2 pixel format <INVALID>
[0:39:29.097535387] [11724]  WARN Formats formats.cpp:997 Unsupported pixel format 0x00000000
[0:39:29.097531498] [11724] ERROR RPI pisp.cpp:228 Pixel format <INVALID> unsupported
[0:39:29.097553276] [11724] FATAL default pisp.cpp:229 assertion "0" failed in toPiSPImageFormat()
Backtrace:
??? [0x00007fff6dd7aa88] (/usr/lib/aarch64-linux-gnu/libcamera.so.0.2.0 [0x00007fff6dd7aa88])
??? [0x00007fff6dd7ad68] (/usr/lib/aarch64-linux-gnu/libcamera.so.0.2.0 [0x00007fff6dd7ad68])
libcamera::PiSPCameraData::platformValidate(libcamera::RPi::RPiCameraConfiguration*) const+0x390 (/usr/lib/aarch64-linux-gnu/libcamera.so.0.2.0 [0x00007fff6dd7b500])
libcamera::RPi::RPiCameraConfiguration::validate()+0x3f4 (/usr/lib/aarch64-linux-gnu/libcamera.so.0.2.0 [0x00007fff6dd728e8])
cv::CvCapture_libcamera_proxy::grabFrame()+0x334 (/usr/local/lib/libopencv_videoio.so.4.7.0 [0x00007fff6f5b5164])
cv::VideoCapture::grab()+0x74 (/usr/local/lib/libopencv_videoio.so.4.7.0 [0x00007fff6f57cf84])
cv::VideoCapture::read(cv::_OutputArray const&)+0x34 (/usr/local/lib/libopencv_videoio.so.4.7.0 [0x00007fff6f57b914])
main+0x120 (/home/pi/devel/libcamvid/vid [0x00005556228712b4])
__libc_start_call_main+0x70 (../sysdeps/nptl/libc_start_call_main.h:74)
__libc_start_main@@GLIBC_2.34+0x98 (../csu/libc-start.c:128)
_start+0x30 (/home/pi/devel/libcamvid/vid [0x00005556228710b0])
Aborted

ml-kalju commented 1 month ago

Hi, any update on this?

advait-0 commented 1 month ago

@swdee the pixelformats used by your camera 'SRGGB10_CSI2P' and 'SRGGB8' aren't supported at the moment :( and I don't have a compatible camera. Ill look into the format and if I find the documentation I'll push an update so then maybe you can test it out. I have updated the Documentation for my branch with usage instructions, a section describing what works and known issues, so its easier for everyone to test it out.

yekni commented 1 month ago

Hi @advait-0, I don't think it's compatible with Python. It's designed for C. Is there a Python version available?

advait-0 commented 3 weeks ago

Hi @yekni, Although most of the documentation is for C++, all the supported stuff as per this also works with the OpenCV Python API. After you have built from source and installed try this sample program.

yekni commented 3 weeks ago

@advait-0 I couldn't find the Python version of it. Is there any link or something to get the Python version?

advait-0 commented 3 weeks ago

@yekni What exactly are you trying to use it for? You have to build and install this from source, the instructions to do so are in its readme file. After installing you can use the python API to some extent.

DominiquePaul commented 2 weeks ago

Any update on this? :)

spikecodes commented 4 days ago

^^

kbingham commented 3 days ago

I hope I'm not replying out of place here, but just to highlight - that the work done here so far was part of a Google summer of code project. That project was limited in time, and finished, so there is no current development on this, and I don't feel that it's fair to 'burden' Advait with pressure on this topic, as the internship has completed. (Of course Advait is free and welcome to continue, but this is not a task 'assigned' to him)

If others would like to contribute then please join in, but as I understand it there is no active development on this topic. I'd love to see it progress too - so please let us know if anyone wants to help out here!

opencv / opencv

Add support for libcamera #21653