[BUG] RGB Colorspaces and opencv

[davidbonn]

Describe the bug

This is really a documentation issue, or I am extremely confused.

opencv by default uses a BGR colorspace, because Windows and Intel architectures. picamera2 often confusingly refers to this as an RGB channel. libcamera appears to do so as well.

Weirder still, in the documentation, you recommend converting a YUV stream to a BGR stream by using the opencv call:

rgb = cv2.cvtColor(yuv420, cv2.COLOR_YUV420p2RGB)

While opencv is using BGR encoding and, in fact, using cv2.COLOR_YUV420p2BGR very much does the Wrong Thing! I don't know what bothers me more, that using RGB does the Right Thing or that I don't understand why or how it could.

To Reproduce This bug is really about confusing documentation.

Expected behaviour The best solution would be an explanation in the documentation of why this is happening and what is going on. Also to add to the confusion libcamera apparently uses RGB when it really means BGR as well.

You can't really "fix" this by changing code, so clarifying it in the documentation seems like the best bet.

Hardware and Software: Pi 4, ov5647 camera, bookworm, stock packages for opencv and picamera2

[davidplowman]

I agree it's confusing. Picamera2 follows libcamera, and I think libcamera follows Linux kernel DRM, or maybe it's media bus codes, I'm not sure. I find the Linux naming really confusing, things all seem to be back to front to me, but I'm sure there's someone out there who could explain why it's correct.

I don't really have a better workaround than to go with what works. I'm afraid I can't explain it either.

[truher]

i spent an hour losing my mind today about this issue.

putting a sentence in the picamera2 manual would really help.

[geezacoleman]

I had this issue today too - was trying to figure out why the output images were inverted when displayed in OpenCV with the selected format as BGR888. Very glad I came across this issue.

So it seems the libcamera RGB888 output is actually BGR format? I.e.:

blue_channel = frame[0]
green_channel = frame[1]
red_channel = frame[2]

If so, a line in the documentation would be fantastic.

[davidplowman]

I thought I'd done this, but I can't find any evidence that I did. Can anyone suggest where in the documentation would be the most useful place? Thanks!

[geezacoleman]

Thanks for the fast response! Upon re-reading 4.2.2.2, pg 21 more closely I can see I didn't read carefully enough, because you have:

• RGB888 - 24 bits per pixel, ordered [B, G, R].
• BGR888 - as above, but ordered [R, G, B].

which makes sense, but it isn't the most obvious. Maybe including a line there under the dot points noting the BGR vs RGB discrepancy and which should be used with OpenCV (just as an example)?

[davidplowman]

I've added a "WARNING" box just below that which explains this. Hopefully the online copy will get updated before too long. Thanks!

[geezacoleman]

After a bit more investigation, I'm still a bit lost and it seems to be due to some underlying driver changes.

For context I'm developing this open-source weed detection system, OpenWeedLocator. I'm in the process of transferring to picamera2 on the picamera2 branch.

On a fresh install of Raspbian 64-bit with no colour transformations with OpenCV, RGB888 results in an off-colour image when displayed with OpenCV imread() - where blue/red have been swapped. But if I run sudo rpi-update, which I was using to see if I could fix an unrelated error with the HQ camera on Port 1, the exact same clone of the code results in a correctly coloured image displayed by OpenCV.

This is the camera instantiation code in video.py:

def __init__(self, src=0, resolution=(416, 320), exp_compensation=-2, **kwargs):
        self.name = 'Picamera2Stream'
        self.size = resolution  # picamera2 uses size instead of resolution, keeping this consistent
        self.frame_width = None
        self.frame_height = None

        # set the picamera2 config and controls. Refer to picamera2 documentation for full explanations:
        #
        self.configurations = {
            # for those checking closely, using RGB888 may seem incorrect, however libcamera means a BGR format. Check
            # https://github.com/raspberrypi/picamera2/issues/848 for full explanation.
            "format": 'RGB888',
            "size": self.size
        }

        self.controls = {
            "AeExposureMode": 1,
            "AwbMode": libcamera.controls.AwbModeEnum.Daylight,
            "ExposureValue": exp_compensation
        }

        # Update config with any additional/overridden parameters
        self.controls.update(kwargs)

        # Initialize the camera
        self.camera = Picamera2(src)
        self.camera_model = self.camera.camera_properties['Model']

        if self.camera_model == 'imx296':
            print('[INFO] Using IMX296 Global Shutter Camera')

        elif self.camera_model == 'imx477':
            print('[INFO] Using IMX477 HQ Camera')

        elif self.camera_model == 'imx708':
            print('[INFO] Using Raspberry Pi Camera Module 3. Setting focal point at 1.2 m...')
            self.controls['AfMode'] = libcamera.controls.AfModeEnum.Manual
            self.controls['LensPosition'] = 1.2

        else:
            print('[INFO] Unrecognised camera module, continuing with default settings.')

        try:
            self.config = self.camera.create_preview_configuration(main=self.configurations,
                                                                   controls=self.controls)
            self.camera.configure(self.config)
            self.camera.start()

Perhaps this is the wrong place to post this. I will start looking through the changes made in the rpi-update, but even so if the fresh install of Raspian with RGB888 image format results in a non-BGR image, then I'm also a bit lost.

Thanks for the help.

[davidplowman]

There was a colour swap problem with the GS cam a while back. Can you check what kernel and firmware versions you have? So the output of uname -a and vcgencmd version.

[davidplowman]

Actually, if sudo rpi-update has fixed the problem, then it suggests the fix still hasn't made it into the "official release" yet, but it should do soon I think.

[geezacoleman]

I was wondering if it was related to this issue. So just to be sure, the correct version is rpi-update with RGB888 resulting in a BGR image? I might have to hold off updating this until this makes the 'official release'. Any idea how long that usually takes?

From the updated/corrected version post rpi-update:

Linux raspberrypi 6.6.28-v8-16k+ #1758 SMP PREEMPT Fri Apr 19 10:40:08 BST 2024 aarch64 GNU/Linux

2024/04/18 09:45:00 
Copyright (c) 2012 Broadcom
version 86ccc427 (release) (embedded)

And from the original version:

Linux raspberrypi 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux

2024/04/18 09:45:00 
Copyright (c) 2012 Broadcom
version 86ccc427 (release) (embedded)

Thanks again for the explanation - was pulling my hair out trying to work out the difference between the two images.

[davidplowman]

Don't think that's the same issue because it seems to be talking about a different device. But colour swaps, especially red-blue ones, well, they occur with depressing frequency tbh. I think an official release of the latest image is imminent, I know I've been asking for it because this isn't the only thing that I know to be broken currently.

You can easily tell if you've got a colour swap by doing self.camera.start(show_preview=True) and it should be very obvious!

[geezacoleman]

Ah righto, I'm not the most familiar with this side of development!

Unfortunately, these are meant to be run without a display on the back of tractors, so can't check everytime, but I might just have to set a specific kernel version in the setup.

[davidplowman]

Sure, I just meant you could use the preview during development while there are issues like this! Hopefully once the fixed image gets released that will be the last of the colour swaps...

Anyway, sounds like a really cool project. Good luck with it!

[geezacoleman]

Ah I see, that makes more sense. Appreciate it! And thanks for all the work you do with picamera2 and the open source community.