Converting android camera images into RGB images

[cyanic-selkie]

Hi! I'm trying to use this library to convert images from android camera into RGB. The format returned is YUV_420_888, meaning 8 bits for each channel.

I am testing this on an image with exported planes and strides from android. I am using the yuv420_to_rgb function like so:

let data = ...;

let y_plane: Vec<u8> = data.yPlane.iter().map(|&x| x as u8).collect();
let u_plane: Vec<u8> = data.uPlane.iter().map(|&x| x as u8).collect();
let v_plane: Vec<u8> = data.vPlane.iter().map(|&x| x as u8).collect();

let mut buffer = vec![0u8; (data.width * data.height * 3) as usize];

yuv420_to_rgb(
    &y_plane,
    data.yStrideRowStride,
    &u_plane,
    data.uStrideRowStride,
    &v_plane,
    data.vStrideRowStride,
    &mut buffer,
    data.width * 3,
    data.width,
    data.height,
    yuvutils_rs::YuvRange::Full,
    yuvutils_rs::YuvStandardMatrix::Bt709,
);

Since the ByteArray in kotlin serializes as i8, I had to cast the planes to u8. Now, the image kind of decodes into a black and white version with spotty colorized pixels. Sort of like the chroma channels are out of alignment.

The image is (640, 480) with all 3 strides being 640 (I was expecting the strides for chroma channels to be half that, especially since I confirmed there are twice as many bytes in the luminance channel than the chroma channels). Since I am very unfamiliar with YUV, I was hoping you might be able to help me debug the issue here.

[awxkee]

Android never uses YUV420 planar format except sometimes emulator on qemu decides to do so. In 100% other base cases you definetely will get NV12 ( it is YUV420 but Bi-planar, not a planar).

Here is exactly mine code how to decide what format actually android sends you:

    private fun getImageFormat(image: ImageProxy): MeasurementImageFormat {
        if (imageFormat == null) {
            imageFormat =
                if (image.format == ImageFormat.NV21 && image.planes.count() >= 2 && image.planes[1].pixelStride == 2) {
                    MeasurementImageFormat.NV21
                } else if (image.format == ImageFormat.YUV_420_888 && image.planes.count() >= 2 && image.planes[1].pixelStride == 2) {
                    MeasurementImageFormat.NV12
                } else if (image.format == ImageFormat.YUV_422_888 && image.planes.count() == 3 && image.planes[1].pixelStride == 1 && image.planes[2].pixelStride == 1) {
                    MeasurementImageFormat.YUV422_PLANAR
                } else if (image.format == ImageFormat.YUV_420_888 && image.planes.count() == 3 && image.planes[1].pixelStride == 1 && image.planes[2].pixelStride == 1) {
                    MeasurementImageFormat.YUV420_PLANAR
                } else if (image.format == ImageFormat.NV16 && image.planes.count() >= 2 && image.planes[1].pixelStride == 2) {
                    MeasurementImageFormat.NV16
                } else if (image.format == ImageFormat.YUV_422_888 && image.planes.count() >= 2 && image.planes[1].pixelStride == 2) {
                    MeasurementImageFormat.NV16
                } else {
                    MeasurementImageFormat.UNKNOWN
                }
            if (imageFormat == MeasurementImageFormat.UNKNOWN) {
                throw something.
            }
        }
        return imageFormat!!
    }

If you're not requesting something specific you'll never get nothing except NV12, and as I mentioned above on qemu sometimes YUV420 planar.

[awxkee]

Also as your stride is equal to image width it is definitely not a yuv420 because in this case it should be half of this.

[cyanic-selkie]

Awesome, thanks! I managed to decode it by concatenating the U and V buffers. Frankly, it looks like android really dropped the ball here with the API.

One question I have left is what are the correct range and matrix values? I can't seem to find any reference to this in the android documentation...

[awxkee]

Awesome, thanks! I managed to decode it by concatenating the U and V buffers. Frankly, it looks like android really dropped the ball here with the API.

You have to pass just plane[1] as UV plane this should work.

One question I have left is what are the correct range and matrix values? I can't seem to find any reference to this in the android documentation...

I also would like to know where such documentation is.

But in common it might be Bt.601 for images small than HD, otherwise if you're not requested high bit depth or HDR Bt.709, or just always Bt.709 except high bit-depth and HDR cases.

[cyanic-selkie]

You have to pass just plane[1] as UV plane this should work.

Hm, if I do that, I get an error: unsafe precondition(s) violated: slice::get_unchecked requires that the index is within the slice

What's more, I checked to make sure U and V planes are not equal.

let data = ...;

let y_plane: Vec<u8> = data.yPlane.iter().map(|&x| x as u8).collect();
let u_plane: Vec<u8> = data.uPlane.iter().map(|&x| x as u8).collect();

let mut buffer = vec![0u8; (data.width * data.height * 3) as usize];
yuv_nv12_to_rgb(
    &y_plane,
    data.yStrideRowStride,
    &u_plane,
    data.uStrideRowStride,
    &mut buffer,
    data.width * 3,
    data.width,
    data.height,
    yuvutils_rs::YuvRange::Full,
    yuvutils_rs::YuvStandardMatrix::Bt709,
);

[awxkee]

Ensure that your UV plane size is

let uv_capacity = uv_stride as usize * (height / 2) as usize;

Android might also some times pass confusing sizes.

Also I might missed to note that Range in cameras is always in TV range ( limited ) unless is stated otherwise

This works just fine.

        mut env: JNIEnv,
        _: jobject,
        y_buffer: jobject,
        y_stride: jint,
        uv_buffer: jobject,
        uv_stride: jint,
        width: jint,
        height: jint,
        isNV21: bool,
    ) -> jobject {
        let y_byte_buffer = JByteBuffer::from(JObject::from_raw(y_buffer));
        let uv_byte_buffer = JByteBuffer::from(JObject::from_raw(uv_buffer));
        let y_result = env.get_direct_buffer_address(&y_byte_buffer);
        let uv_result = env.get_direct_buffer_address(&uv_byte_buffer);
        if let (Ok(y_buffer), Ok(uv_buffer)) = (y_result, uv_result) {
            let y_capacity = y_stride as usize * (height / 2) as usize;
            let uv_capacity = uv_stride as usize * (height / 2) as usize;
            let y_slice = unsafe { slice::from_raw_parts(y_buffer as *const u8, y_capacity) };
            let uv_slice = unsafe { slice::from_raw_parts(uv_buffer as *const u8, uv_capacity) };
                yuv_nv12_to_bgra(
                    y_slice,
                    y_stride as u32,
                    uv_slice,
                    uv_stride as u32,
                    &mut bgra_buffer,
                    bgra_stride as u32,
                    width as u32,
                    height as u32,
                    YuvRange::TV,
                    YuvStandardMatrix::Bt601,
                );

[cyanic-selkie]

I'll close this issue as I have a working solution. However, I would appreciate it if you cleared that last part up for me.

My solution doesn't change the capacity of the Y plane the way you did. In fact, it crashes it with the index out of bounds error. If I leave the Y plane the way it is, it all works out. What gives?

[awxkee]

I don't remember this exactly, only thing I exact remember that android or jni is messing with providing actual ByteBuffer size, and you have to force it to it real size.

Also, note this works only for even sized images, if you're not ensured that size is always even, then use uv_stride as usize * ((height + 1) / 2) as usize for reshape and same for luma plane.