Unable to get camera frame on macbook

[bokutotu]

Thank you for creating such a great repository! I have a issue when using Mabbook pro 14inc (M1 max). Below are the steps to reproduce the error.

Ventura 13.1 machine MacBook Pro 14-inch 2021 Apple M1 Max rust version 1.66.0 Cargo.toml

name = "show_image"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
show-image="0.13.1"
image = { version="0.24.5" }
nokhwa = { version = "0.10.3", features=["input-native", "output-threaded"] }

Code (main.rs)

use nokhwa::{
    nokhwa_initialize,
    pixel_format::{RgbAFormat, RgbFormat},
    query,
    utils::{ApiBackend, RequestedFormat, RequestedFormatType},
    threaded::CallbackCamera
};

fn main() {
    // only needs to be run on OSX
    nokhwa_initialize(|granted| {
        println!("User said {}", granted);
    });
    let cameras = query(ApiBackend::Auto).unwrap();
    cameras.iter().for_each(|cam| println!("{:?}", cam));

    let format = RequestedFormat::new::<RgbFormat>(RequestedFormatType::AbsoluteHighestFrameRate);

    let first_camera = cameras.first().unwrap();

    let mut threaded = CallbackCamera::new(first_camera.index().clone(), format, |buffer| {
        let image = buffer.decode_image::<RgbAFormat>().unwrap();
        println!("{}x{} {}", image.width(), image.height(), image.len());
    })
    .unwrap();
    threaded.open_stream().unwrap();
    #[allow(clippy::empty_loop)] // keep it running
    loop {
        let frame = threaded.poll_frame().unwrap();
        let image = frame.decode_image::<RgbAFormat>().unwrap();
        println!(
            "{}x{} {} naripoggers",
            image.width(),
            image.height(),
            image.len()
        );
    }
}

Error

User said true
CameraInfo { human_name: "FaceTime HD Camera", description: "Apple Inc.: FaceTime HD Camera - AVCaptureDeviceTypeBuiltInWideAngleCamera, Unspecified f0", misc: "47B4B64B-7067-4B9C-AD2B-AE273A71F4B5", index: Index(0) }
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ProcessFrameError { src: NV12, destination: "RGB", error: "bad input buffer size" }', src/main.rs:63:56
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: ProcessFrameError { src: NV12, destination: "RGB", error: "bad input buffer size" }', src/main.rs:55:57

If not already resolved, I would like to contribute.

[bookshiyi]

https://github.com/l1npengtul/nokhwa/issues/72#issuecomment-1381067454 Source frame format NV12 may not be supported on macintosh untill now, I guess.

[l1npengtul]

It is likely a fault of the library improperly setting FourCC

[hallucinogen]

I found that the first problem is the NV12 checker in this line: https://github.com/l1npengtul/nokhwa/blob/ad93e0a6c521f42424e7ec717455eece9f3e1fd2/nokhwa-core/src/types.rs#L1445

if data.len() != ((resolution.width() * resolution.height() * 3) / 2) as usize {

I tried to change it into

if data.len() != (resolution.width() * resolution.height() * 2) as usize {

Now it passes the decoding round.

But the resulting decoded image is faulty. It looks like it's split in the middle, and then we put the left part on the right and the right part on the left. I feel I am close to a solution, but can't really point it out

[vaqxai]

Does not work on Macbook Air 2014 13". Same error.

[stolinski]

Anyone have a work around for this? All cams in macOS are coming in as NV12 and hitting the same bad input buffer size issue.

[anselanza]

Damn, I just hit this problem, too.

Until NV12 support comes in v0.11, it's possible to use a crate like DCV Color Primitives to do the necessary conversion.

[bluezheng]

Hi, is there any workaround for this issue?

[marcpabst]

Hi, I'm seeing this issue as well. I think the problem is that AVfoundation returns frames in UYVY format (https://wiki.videolan.org/YUV#UYVY), even if you tell nokhwa to request NV12 (or some other format).

Here is my current workaround to deal with this kind of data (note that this only works on Rust nightly). Alternativly, you could split the interleaved planes and use the aforementioned dcv-color-primitives crate for conversion.

#![feature(portable_simd)]
use std::simd::SimdFloat;
use std::simd::f32x4;
use rayon::prelude::*;

#[inline]
pub fn uyvy_to_rgb24(in_buf: &[u8], out_buf: &mut [u8]) {
    debug_assert!(out_buf.len() as f32 == in_buf.len() as f32 * 1.5);

    in_buf
        .par_chunks_exact(4) // FIXME: use par_array_chunks() when stabalized (https://github.com/rayon-rs/rayon/pull/789)
        .zip(out_buf.par_chunks_exact_mut(6))
        .for_each(|(ch, out)| {
            let y1 = ch[1];
            let y2 = ch[3];
            let cb = ch[0];
            let cr = ch[2];

            let (r, g, b) = ycbcr_to_rgb(y1, cb, cr);

            out[0] = r;
            out[1] = g;
            out[2] = b;

            let (r, g, b) = ycbcr_to_rgb(y2, cb, cr);

            out[3] = r;
            out[4] = g;
            out[5] = b;
        });
}

// COLOR CONVERSION: https://stackoverflow.com/questions/28079010/rgb-to-ycbcr-using-simd-vectors-lose-some-data

#[inline]
fn ycbcr_to_rgb(y: u8, cb: u8, cr: u8) -> (u8, u8, u8) {
    let ycbcr = f32x4::from_array([y as f32, cb as f32 - 128.0f32, cr as f32 - 128.0f32, 0.0]);

    // rec 709: https://mymusing.co/bt-709-yuv-to-rgb-conversion-color/
    let r = (ycbcr * f32x4::from_array([1.0, 0.00000, 1.5748, 0.0])).reduce_sum();
    let g = (ycbcr * f32x4::from_array([1.0, -0.187324, -0.468124, 0.0])).reduce_sum();
    let b = (ycbcr * f32x4::from_array([1.0, 1.8556, 0.00000, 0.0])).reduce_sum();

    (clamp(r), clamp(g), clamp(b))
}

#[inline]
fn clamp(val: f32) -> u8 {
    if val < 0.0 {
        0
    } else if val > 255.0 {
        255
    } else {
        val.round() as u8
    }
}

(adapted from here: https://gist.github.com/arifd/ea820ec97265a023e67a88b66955855d)

As a sidenote, it looks like recent MacOS versions do not support transparent raw access to the camera outputs anymore, i.e. you will always be offered uyvy422, yuyv422, nv12, 0rgb, and bgr0, regardless of what the camera actually supports. MJPEG streams will still work when requesting a resolution/fps mode only supported through MJPEG (as is common for higher resolutions on UVC cameras , but AVfoundation will handle the conversion to one of the listed pixel formats internally.

@l1npengtul, not sure what the current status is here (I'm a bit confused with the different branches) but I might be able to help you with debugging this.

Edit: Forgot to say that this is on a M2 MacBook Pro on Ventura 13.4.

[yamt]

Hi, I'm seeing this issue as well. I think the problem is that AVfoundation returns frames in UYVY format (https://wiki.videolan.org/YUV#UYVY), even if you tell nokhwa to request NV12 (or some other format).

see https://github.com/l1npengtul/nokhwa/pull/151

[Moon1102]

@l1npengtul Could you take a look at the code below when you have a moment? This is a test case I wrote where I was able to convert a camera data frame into an image and save it, on Rust v1.79, MacOS 13.5.2 (Apple M1). The only problem now is that the color is bluish, I don't know why yet, this code works well and the color is correct when I use the av-foundation library.

#[cfg(test)]
mod nokhwa_camera_macos {
    use image::{ImageBuffer, Rgb};
    use nokhwa::{Buffer, nokhwa_initialize, NokhwaError, pixel_format::RgbFormat, query, threaded::CallbackCamera, utils::{ApiBackend, RequestedFormat, RequestedFormatType}};
    use nokhwa::utils::Resolution;

    #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
    async fn test() -> Result<(), NokhwaError> {
        // only needs to be run on OSX
        nokhwa_initialize(|granted| {
            println!("User said {}", granted);
        });
        let cameras = query(ApiBackend::AVFoundation)?;

        let format = RequestedFormat::new::<RgbFormat>(RequestedFormatType::AbsoluteHighestFrameRate);

        let first_camera = cameras.first().ok_or(NokhwaError::GeneralError("cannot find camera".to_string()))?;

        let mut threaded = CallbackCamera::new(first_camera.index().clone(), format, |buffer| {
            convert_buffer_to_image(buffer).map(|buf|
                buf.save("src/output/output.png")
            ).ok();
        })?;
        threaded.open_stream()?;
        #[allow(clippy::empty_loop)] // keep it running
        loop {
            threaded.poll_frame()?;
        }
    }

    fn convert_buffer_to_image(buffer: Buffer) -> Result<ImageBuffer<Rgb<u8>, Vec<u8>>, NokhwaError> {
        let Resolution { width_x: width, height_y: height } = buffer.resolution();
        let mut image_buffer = ImageBuffer::<Rgb<u8>, Vec<u8>>::new(width, height);
        let data = buffer.buffer();

        for (y, chunk) in data.chunks_exact((width * 2) as usize).enumerate().take(height as usize) {
            for (x, pixel) in chunk.chunks_exact(4).enumerate() {
                let [v, y2, u, y1] = [pixel[0] as f32, pixel[1] as f32, pixel[2] as f32, pixel[3] as f32];
                let x = (x * 2) as u32;
                image_buffer.put_pixel(x, y as u32, yuv_to_rgb(y1, u, v));
                image_buffer.put_pixel(x + 1, y as u32, yuv_to_rgb(y2, u, v));
            }
        }

        Ok(image_buffer)
    }

    //YUV to RGB conversion BT.709
    fn yuv_to_rgb(y: f32, u: f32, v: f32) -> Rgb<u8> {
        let r = y + 1.5748 * (v - 128.0);
        let g = y - 0.1873 * (u - 128.0) - 0.4681 * (v - 128.0);
        let b = y + 1.8556 * (u - 128.0);

        Rgb([r as u8, g as u8, b as u8])
    }
}

[l1npengtul]

Honestly, the current avfoundation code is in such a state that Ive blanked it out of my mind. I really cant help you here, unfortunately.

I'll work on making nokhwa itself use avfoundation.

[Moon1102]

Okay, I just wanted to say that I found a way to start by replacing the code let image = buffer.decode_image::<RgbAFormat>().unwrap(); temporarily with my code, so that I can successfully start using nokhwa, otherwise I would have to accept the NV12: bad input buffer size error. I can't say I've managed to solve the problem completely, but it's going in a good direction for me~ I'll take another look at how to fix the bluish pictures sometime soon.

[Moon1102]

Okay, I just wanted to say that I found a way to start by replacing the code let image = buffer.decode_image::<RgbAFormat>().unwrap(); temporarily with my code, so that I can successfully start using nokhwa, otherwise I would have to accept the NV12: bad input buffer size error. I can't say I've managed to solve the problem completely, but it's going in a good direction for me~ I'll take another look at how to fix the bluish pictures sometime soon.

Just change let [v, y2, u, y1] = [pixel[0] as f32, pixel[1] as f32, pixel[2] as f32, pixel[3] as f32]; to let [u, y1, v, y2] = [pixel[0] as f32, pixel[1] as f32, pixel[2] as f32, pixel[3] as f32]; so you can get the image with the correct color.

[raphaelmenges]

Just change let [v, y2, u, y1] = [pixel[0] as f32, pixel[1] as f32, pixel[2] as f32, pixel[3] as f32]; to let [u, y1, v, y2] = [pixel[0] as f32, pixel[1] as f32, pixel[2] as f32, pixel[3] as f32]; so you can get the image with the correct color.

Thanks @Moon1102 for the proposed workaround. It works on my MacBook for the integrated camera, with M1 CPU and macOS 14.5! Do you know whether this affects only Macs with Apple Silicon or all Macs? Or does it depend on the camera?

[Moon1102]

@raphaelmenges Sorry, at the moment I only have a MacBook with an Apple Silicon M1 and an integrated camera, so I'm not sure if it will work properly on other types of Macs. I think the code for nokhwa's codecs may need further improvement in terms of adapting to individual devices now, perhaps will better after version 0.11. Also, I'm curious if you're experiencing error #165? i haven't found a solution yet.

[paviro]

@Moon1102 using your code I still get this result: Any idea? This is a USB capture card connected to the Mac itself.

[Moon1102]

@Moon1102 using your code I still get this result: Any idea? This is a USB capture card connected to the Mac itself.

@paviro Hi. My camera captures video frames of type kCVPixelFormatType_422YpCbCr8 aka "2vuy". There are various types of them, which you can refer to it to determine the type of your own video frames. And this example can be used to analyze and determine your video frame type.

[paviro]

Thank you for the tips @Moon1102 will investigate further!

[paviro]

@Moon1102 My camera also seems to output 2vuy. Not sure what I am doing wrong? Could you give me any more hints? Totally fine if you don't want to sadly I am a bit new to rust and the video world.

pixel buffer: "<CVPixelBuffer 0x600001c8f4f0 width=1920 height=1080 bytesPerRow=3840 pixelFormat=2vuy iosurface=0x6000029ecf10 surfaceid=2146 attributes={
    Height = 1080;
    IOSurfaceProperties =     {
        IOSurfacePurgeWhenNotInUse = 1;
    };
    PixelFormatType = 846624121;
    Width = 1920;
} propagatedAttachments={
    CGColorSpace = "<CGColorSpace 0x600000fed3e0> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; Composite NTSC)";
    CVImageBufferColorPrimaries = "SMPTE_C";
    CVImageBufferTransferFunction = "ITU_R_709_2";
    CVImageBufferYCbCrMatrix = "ITU_R_601_4";
} nonPropagatedAttachments={
}>"

[paviro]

Well for now I just patched raw_fcc_to_frameformat in nokhwa-bindings-macos/src/lib.rs to return FrameFormat::YUYV for 875704438 instead of FrameFormat::NV12.

fn raw_fcc_to_frameformat(raw: OSType) -> Option<FrameFormat> {
        match raw {
            kCMVideoCodecType_422YpCbCr8 | kCMPixelFormat_422YpCbCr8_yuvs => {
                Some(FrameFormat::YUYV)
            }
            kCMVideoCodecType_JPEG | kCMVideoCodecType_JPEG_OpenDML => Some(FrameFormat::MJPEG),
            kCMPixelFormat_8IndexedGray_WhiteIsZero => Some(FrameFormat::GRAY),
            kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange
            | kCVPixelFormatType_420YpCbCr8BiPlanarFullRange
            | 875704438 => Some(FrameFormat::YUYV),
            kCMPixelFormat_24RGB => Some(FrameFormat::RAWRGB),
            _ => None,
        }
    }

Now simply doing this works without any extra steps and result in a correct image:

let frame = camera.frame()?;
let decoded = frame.decode_image::<RgbFormat>().unwrap();

let output_path = Path::new("output.jpg");
decoded.save(output_path)?;

[Moon1102]

@Moon1102 My camera also seems to output 2vuy. Not sure what I am doing wrong? Could you give me any more hints? Totally fine if you don't want to sadly I am a bit new to rust and the video world.

pixel buffer: "<CVPixelBuffer 0x600001c8f4f0 width=1920 height=1080 bytesPerRow=3840 pixelFormat=2vuy iosurface=0x6000029ecf10 surfaceid=2146 attributes={
    Height = 1080;
    IOSurfaceProperties =     {
        IOSurfacePurgeWhenNotInUse = 1;
    };
    PixelFormatType = 846624121;
    Width = 1920;
} propagatedAttachments={
    CGColorSpace = "<CGColorSpace 0x600000fed3e0> (kCGColorSpaceICCBased; kCGColorSpaceModelRGB; Composite NTSC)";
    CVImageBufferColorPrimaries = "SMPTE_C";
    CVImageBufferTransferFunction = "ITU_R_709_2";
    CVImageBufferYCbCrMatrix = "ITU_R_601_4";
} nonPropagatedAttachments={
}>"

@paviro Sorry, I just saw the message and found that you've already solved the problem in a simpler and better way. That's awesome. I debugged it and got a format value of 846624121, which is different from yours. If your value is 875704438, that corresponds to kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange and it doesn't matter anymore.

[paviro]

I don't fully understand as you can see the av-foundation binding does return 846624121 but somehow the same camera in nokhwa seems to return 875704438. Anyhow glad this works for now! Thank you for your support!