search

waveform80 / picamera

A pure Python interface to the Raspberry Pi camera module
https://picamera.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
1.57k stars 357 forks source link

What is the real format of the uncompressed RGB and RGBA raw data outputs? #659

Open darkcurrent opened 3 years ago

darkcurrent commented 3 years ago

I want to record RAW RGB888 low resolution video. In the docs you stated that camera outputs YUV format before the data comes to the video splitter. Based on this information, is the video in YUV420 format at the camera output? If so I think it is not possible to convert YUV420 to RGB888 without any data loss due to downsampled YUV channels? Am I right?

6by9 commented 3 years ago

It's perfectly possible to convert from YUV 4:2:0 to RGB.

Yes there is a very small amount of loss of colour detail from the chroma planes having been subsampled, but they will have been upsampled generally using interpolation during the conversion. There will also be a very small amount of rounding that occurs due to the multiplications in the conversion matrices.

It still doesn't make the output anything other than RGB 888 or RGB[A|X]8888 though. Otherwise you'll say that the output of a JPEG decoder isn't YUV - it is, but has been through a lossy conversion path. (I'm ignoring the fact that RGB JPEGs exist too).

darkcurrent commented 3 years ago

@6by9 Thank you!

Unfortunately interpolation doesn't work for me because I'm already trying to observe very small amount of changes in the color. So the non-interpolated real RGB888 encoded video is some kind of necessity for my application. In this point I have two more kind questions if you let me:

1) I am using the latest version (1.13) but in the previous documentation it is stated that at the camera output mode for the video port is fixed to YUV (marked blue in the attached image below) in order to make the encoder operations faster. What is the actual raw video format at this point? (YUV420, YUV444 or etc?) I am asking this because if it is YUV444 I can convert it to RGB888 and use it, otherwise some very small amount of changes will be loss.

2) In the same documentation it is stated that resizer component does this YUV to RGB conversions. If first questions answer is "YUV420", does the resizer convert YUV420 to RGB888 by using interpolation?

alt text

6by9 commented 3 years ago

The source sensor has a Bayer colour filter at the front of it, therefore your colour channels have already been interpolated as part of the demosaicing step.

Within the firmware camera stack, the output of the ISP is ALWAYS YUV420 or YUV422. If you ask for RGB on a camera port then that will be converted in software at point where an output buffer is being filled. A number of the postprocessing stages were only ever written to work on YUV data, therefore it made more sense to do this conversion than try updating all of those to work in the RGB domain as well.

I'm not going to dig into the YUV to RGB conversion code, largely because there are a number of other parameters at play. The subsampled chroma sample can be sampled at any of a number of locations, known as the chroma siting. See https://en.wikipedia.org/wiki/Chroma_subsampling#4:2:0 for a basic description. For each of the horizontal and vertical directions the sample can be taken on top of a luma pixel (co-sited), or between two samples (interstitial). I'd have to check out which flavour the ISP produces first to check that every stage was correct, and I'm afraid I just don't have the time to do that.

libcamera does allow requesting RGB direct from the ISP, or you could compare the raw pixel values off the sensor.