RPI4: Libcamera integration CSI+ISP

rsglobal commented 2 years ago

We've made a good progress towards libcamera integration, preview and shots are working, still it require more testing / bug fixing.

Some of links:

Discussions with the maintainers:

https://github.com/kbingham/libcamera/issues/31 https://github.com/kbingham/libcamera/pull/26

Local branches

Configuration:

ADD:

Minigbm changes:

diff --git a/gbm_mesa_driver/gbm_mesa_internals.cpp b/gbm_mesa_driver/gbm_mesa_internals.cpp
index 2f8b0d1be9cc..4ac468a39179 100644
--- a/gbm_mesa_driver/gbm_mesa_internals.cpp
+++ b/gbm_mesa_driver/gbm_mesa_internals.cpp
@@ -321,6 +321,12 @@ int gbm_mesa_bo_create(struct bo *bo, uint32_t width, uint32_t height, uint32_t

        bool scanout = (use_flags & BO_USE_SCANOUT) != 0;
        bool linear = (use_flags & BO_USE_SW_MASK) != 0;
+
+       if (use_flags & (BO_USE_CAMERA_READ | BO_USE_CAMERA_WRITE)) {
+               scanout = true;
+               width = ALIGN(width, 32);
+       }
+
        uint32_t s_format = format;
        int s_height = height;
        if (get_gbm_mesa_format(format) == 0) {

ADD2: RPI4 kernel at least @ v5.10.77 is required:

rsglobal commented 2 years ago

@KonstaT,

Hi,

I've posted it here in case you would like to try it out in your builds.

KonstaT commented 2 years ago

Thanks. I'll surely give it a test and report back. Probably won't be until next week as I'm away from the keyboard for the rest of the week.

KonstaT commented 2 years ago

I tested this on Raspberry Pi 4 with 5.10.78 kernel and Pi camera modules v1 (OV5647) and v2 (IMX219). It seems to work well for camera preview and photos with the stock camera app with both camera hw. Didn't find it to be too unstable or crash (might have missed some though as the logging is highly verbose currently). Camcorder preview shows garbled image and capturing videos doesn't work. Only third party camera app I tested was OpenCamera and camera preview and photos don't work with that (shows same garbled image as stock app with camcorder).

Some notes.

With recent firmware you can set camera_auto_detect=1 in config.txt which is handy if you want to support more than one camera hardware. Also for some reason just setting e.g. dtoverlay=imx219 didn't seem to work so I wonder what else the firmware enables with auto detect.

My camera_hal.yaml looks like this to support camera modules v1, v2 & HD.

cameras:
  "/base/soc/i2c0mux/i2c@1/imx219@10":
    location: back
    rotation: 0
  "/base/soc/i2c0mux/i2c@1/imx477@1a":
    location: back
    rotation: 0
  "/base/soc/i2c0mux/i2c@1/ov5647@36":
    location: back
    rotation: 0

If you move /vendor/etc/libcamera/ipa_rpi.so.sign to /vendor/lib(64)/ (same place as ipa_rpi.so) you get from D libcamera: DEBUG IPAModule ipa_module.cpp:320 ipa_rpi.so: IPA module /vendor/lib64/libcamera//ipa_rpi.so is not signed to D libcamera: DEBUG IPAModule ipa_module.cpp:329 ipa_rpi.so: IPA module /vendor/lib64/libcamera//ipa_rpi.so is signed Untested if this resolves the issue you had with IPA signature.

If you wasn't aware I have Android fork of v4l-utils here (https://github.com/lineage-rpi/android_external_v4l-utils/tree/lineage-19.0). Especially v4l2-ctl comes handy when working with cameras and hw video dec/enc. E.g. to check which dev nodes belong to which driver (to set permissions).

console:/ # v4l2-ctl --list-devices                                            
bcm2835-codec-decode (platform:bcm2835-codec):
        /dev/video10
        /dev/video11
        /dev/video12
        /dev/video18
        /dev/media0

bcm2835-isp (platform:bcm2835-isp):
        /dev/video13
        /dev/video14
        /dev/video15
        /dev/video16
        /dev/media1

unicam (platform:fe801000.csi):
        /dev/video0
        /dev/video1
        /dev/media2

IMO overall this looks very promising so great job as always! :)

On somewhat related matters, there's was a report from an user with external USB camera with [11-09 22:33:57.428 263:263 E/[minigbm:CrosGralloc4Allocator.cc(58)]] Unsupported combination -- pixel format: PixelFormat::YV12, drm format:DRM_FOURCC_9997, usage: BufferUsage::CAMERA_OUTPUT|BufferUsage::COMPOSER_OVERLAY|BufferUsage::CPU_WRITE_OFTEN|BufferUsage::GPU_TEXTURE I assume some DRM_FORMAT_YVU420_ANDROID handling in minigbm's gbm_mesa platform would sort this? I currently don't have any USB webcams that are supported with the AOSP external camera HAL that I can use to test.

Also would https://android.googlesource.com/platform/external/minigbm/+/3a171db9be34ef1e9ff62e3ee17bc0632cf4cdc7 and https://android.googlesource.com/platform/external/minigbm/+/71bc665179f9e8a94c4b2293f6b3406887f96d34 be something that we'd want for gbm_mesa?

rsglobal commented 2 years ago

Thanks for all your comments. It was very useful. I'll update GloDroid with more cams support.

I assume some DRM_FORMAT_YVU420_ANDROID handling in minigbm's gbm_mesa platform would sort this?

I'll test it soon.

Also would https://android.googlesource.com/platform/external/minigbm/+/3a171db9be34ef1e9ff62e3ee17bc0632cf4cdc7 and https://android.googlesource.com/platform/external/minigbm/+/71bc665179f9e8a94c4b2293f6b3406887f96d34 be something that we'd want for gbm_mesa?

Currently I'm running GD through all those CTS test and will add all necessary handling. Also I plan to continuously rebase onto upstream minigbm, so all new patches will enter into my fork sooner or later.

Just in case you also want to play CTS&VTS game, I started to accumulate some experience here: https://github.com/GloDroid/glodroid_manifest/wiki/VTS-&-CTS

rsglobal commented 2 years ago

I redesigned minigbm to support size alignment. Setting size_alignment to 4096 (page) fixed camera crashes. (available in minigbm-next branch). I assume it may also help you with codecs issues.

rsglobal commented 2 years ago

Preview and shots are working. Will be available in v0.7.0.

rsglobal commented 2 years ago

@KonstaT ,

I used info form dts/overlays/* to extract nodes into .yaml file for remaining cameras. Not tested as I do not have them:

https://github.com/GloDroid/glodroid_device/commit/be6ba9a457c28a70f1307c69639a7caeecad0b22#diff-b34d0c03657ef9f5b48562515ea1c69544921cb8e1beb112f793781d899da5d6

KonstaT commented 2 years ago

Thanks. Yep, that's where I got them for the Pi camera modules as well. I'm sticking with the official modules for now but I'll probably add the configs/kernel drivers for the other hw supported with libcamera later as well. I don't have any other hardware to test myself either.

I shipped libcamera with my LineageOS 19 build today so I'm sure people will complain if it's utterly broken. I was very clever again and reverted your logging change which apparently killed all logs for libcamera so there won't be any logs to help, though. :P

GloDroid / glodroid_manifest