How to use nokhwa together with iced

[DanielMSchmidt]

Hi there 👋

I'm trying to use nokhwa (built from this main branch) to show my webcam video in an iced application. I am pretty new to this ecosystem, so sorry if this is a dumb question, but how do I use the camera output with the image widget of iced.

My application looks like this:

use iced::widget::{column, image, text};
use iced::{executor, Subscription};
use iced::{Alignment, Application, Command, Element, Settings, Theme};
use nokhwa::pixel_format::RgbFormat;
use nokhwa::utils::{ApiBackend, CameraInfo, RequestedFormat, RequestedFormatType};
use nokhwa::{query, Camera};

// TODO: set these via environment variables
const CAMERA_QUERY_INTERVAL: u64 = 2000;

pub fn main() -> iced::Result {
    MediaHub::run(Settings::default())
}

struct MediaHub {
    cameras: Box<Vec<CameraInfo>>,
}

#[derive(Debug, Clone)]
enum Message {
    CamerasListed(Box<Vec<CameraInfo>>),
}

fn query_all_cameras() -> Box<Vec<CameraInfo>> {
    Box::new(query(ApiBackend::Auto).unwrap())
}

impl Application for MediaHub {
    type Message = Message;
    type Executor = executor::Default;
    type Flags = ();
    type Theme = Theme;

    fn new(_flags: ()) -> (Self, Command<Self::Message>) {
        (
            Self {
                cameras: query_all_cameras(),
            },
            Command::none(),
        )
    }

    fn update(&mut self, message: Message) -> Command<Self::Message> {
        match message {
            Message::CamerasListed(new_cams) => self.cameras = new_cams,
        }
        Command::none()
    }

    fn subscription(&self) -> Subscription<Message> {
        iced::time::every(std::time::Duration::from_millis(CAMERA_QUERY_INTERVAL))
            .map(|_| Message::CamerasListed(query_all_cameras()))
    }

    fn title(&self) -> String {
        String::from("MediaHub - Iced")
    }

    fn view(&self) -> Element<Message> {
        if self.cameras.len() > 0 {
            let camera_info = self
                .cameras
                .iter()
                .map(|cam| cam.human_name())
                .collect::<Vec<String>>()
                .join(", ");

            let requested = RequestedFormat::new(RequestedFormatType::AbsoluteHighestFrameRate);

            let cam = Camera::new(nokhwa::utils::CameraIndex::Index(0), requested).unwrap();
            let frame = cam.frame().unwrap().decode_image().unwrap();

            let img = image::Handle::from_pixels(frame.width(), frame.height(), frame); // the last argument is wrong, it throws an error

            column![text(camera_info).size(50), image::viewer(img)]
                .padding(20)
                .align_items(Alignment::Center)
                .into()
        } else {
            column![text("No camera found").size(50)]
                .padding(20)
                .align_items(Alignment::Center)
                .into()
        }
    }
}

I can contribute an example for this integration if you like :)

[l1npengtul]

Camera should be created once, and for your application a CallbackCamera would be appropriate to not block the main thread waiting on a frame. Once you get a buffer, you should decode it using RgbFormat to get a ImageBuffer<Rgb<u8> Vec<u8>>, which should be trivial to display in any image and/or canvas2d widget.

[l1npengtul]

senpai is pretty close to release, just need to fix nv12 and make sure AVFoundation works.

[bookshiyi]

senpai is pretty close to release, just need to fix nv12 and make sure AVFoundation works.

When I use nokhwa 0.10.3, the following error is prompted. Is the conversion of nv12 to rgb format still not available?

Captured Single Frame of 4147200
Frame format: NV12
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ProcessFrameError { src: NV12, destination: "RGB", error: "bad input buffer size" }', src/main.rs:97:53

Thank you sincerely.

[l1npengtul]

The NV12 decoder is likely still somewhat buggy, or nokhwa is confused. Try running it through yuyv and see what happens.

If issue persists please open a new issue @bookshiyi