search

luxonis / depthai

DepthAI Python API utilities, examples, and tutorials.
https://docs.luxonis.com
MIT License
936 stars 232 forks source link

[Feature-Request] OAK with microphone; noise cancellation #688

Open rsing92 opened 2 years ago

rsing92 commented 2 years ago

I'm using my OAK-1/OAK-D as a webcam when I'm not using for development. Do you have a product, or plan to have one with microphone so it can be used as a webcam with audio? I couldn't find in store. I'm also looking into noise cancellation capabilities. Do you have or plan to have a node which does noise cancellation? With the current hardware it would be good enough for me to capture audio from laptop and send it to the device and do noise cancellation (which is a blackbox to me). A camera with stereo audio and noise cancellation would make an awesome webcam, when not using for development. Currently pairing the awesome image quality with bad microphones from my laptop results in a bad experience.

Luxonis-Brandon commented 2 years ago

Thanks for the interest and for purchasing and using OAK products!

A couple options, some that are in existence, and some that don't exist yet:

  1. OPAL. Just works. If you want a high-end camera with noise cancellation, (more than) stereo microphone, and a bunch of nice AI features, they're who we recommend: https://opalcamera.com/

This will do all that you're looking for out of the box I think. So it's for folks who don't want to tinker/etc. and want a plug&play solution.

  1. OAK-D-IoT-40 Series 2. This is the opposite. This is more for continuing the development-path that can then be re-purposed when desired to do webcam things. https://github.com/luxonis/depthai-hardware/issues/91

image

It has 6 microphones, a speaker, and a bunch of other stuff. But it doesn't come with any of the features you're looking for (noise cancellation, stereo, etc.) as it's all just the raw product connected with DepthAI API such that any of that could be developed on top of DepthAI (including multi-microphone noise beamforming for noise-cancellation, AI-based noise cancellation, etc.).

So where 1 is available now and plug/play, 2 will be available in some handful of months and then will be again more like OAK-1/OAK-D for developing custom things on top of the API.

Thoughts?

Thanks, Brandon

rsing92 commented 2 years ago

Thanks @Luxonis-Brandon, will look for that. When fetching upstream changes from github for depthai-python I saw this: https://github.com/luxonis/depthai-core/tree/custom_uvc_uac_scripting branch. It seems like it has UAC and UVC support, if I order OAK-D IOT series 2 will I be able to get audio through UAC interface using that? Also are you planning to update the UVC depthai library? Currently it's stuck on 2.13.3, it would be great to have the latest.

Luxonis-Brandon commented 2 years ago

Yes. That's right. We do have this initial support for UAC and UVC at the same time. And that's exactly right.

Yes, we are planning to update and merge it into mainline. Unsure on ETA however though.

rsing92 commented 2 years ago

@Luxonis-Brandon thanks! Who would be the best to ask for an ETA?

Luxonis-Brandon commented 2 years ago

Likely @alex-luxonis actually. He may already know the timing on this. We're trying to get multi-cam support and image manip refactor merged at priority, as those are bigger blockers.

rsing92 commented 2 years ago

Hello again @Luxonis-Brandon May I get an answer, just so I know how to plan ahead? I'm not familiar with multi-cam support and image manip refactor, I didn't see them in latest release notes. I understand if those are blockers for this effort, I would just like to know a rough ETA.

Luxonis-Brandon commented 2 years ago

So I think this is being integrated in with several larger merges. We unfortunately lost an employee because of the Russian invasion of Ukraine (the person is fine, just can't work here anymore - unfortunate situation) which could result in slowing this down a bit.

So I'm not 100% sure on the mainlining on this. @themarpe would have a better idea, but we're still triaging a bit from the loss.

I'd say maybe to plan on mainlining in 2 months. But themarpe may have a better guess.

rsing92 commented 2 years ago

Hello @Luxonis-Brandon I saw the following commit: https://github.com/luxonis/depthai-core/commit/cf91fcb38f4a3e162852df9d72e33057e4fd77c8 on audioproc branch. Does that mean that I can use noise cancellation now? Do you have an example perhaps? I'm eager to try this out even if it's beta.

alex-luxonis commented 2 years ago

@rsing92 We have this feature in the works.

What OAK device do you have? Just to check if/what microphones it has, and if this would be applicable.

Note this feature mainly would apply for devices where at least 3 microphones are present: 2 main mics for recording, and a back mic going as reference input for noise cancellation.

rsing92 commented 2 years ago

@alex-luxonis I don't have any boards that have 3 microphones. I would like to try with data from laptop with XLinkIn, as mentioned here: https://github.com/luxonis/depthai/issues/688#issue-1209680704

Which board do you recommend from your shop to buy for noise cancelling? Do you have performance metrics/evaluation for noise cancellation?

rsing92 commented 2 years ago

@alex-luxonis I don't have any boards that have 3 microphones. I would like to try with data from laptop with XLinkIn, as mentioned here: #688 (comment)

Which board do you recommend from your shop to buy for noise cancelling? Do you have performance metrics/evaluation for noise cancellation?

Do you know the answer on the above @Luxonis-Brandon ?

Luxonis-Brandon commented 2 years ago

Oh ya sorry about the delay. This one has microphones on it: https://shop.luxonis.com/collections/early-access/products/oak-d-iot-40-pro

We haven't mainlined microphone support though. And don't know the eta on doing so.

rsing92 commented 2 years ago

Any update on microphones/noise cancellation? Can I use latest audioproc branch @alex-luxonis ?

rsing92 commented 2 years ago

Hello. Any update? @Luxonis-Brandon @Luxonis-Brian @alex-luxonis @szabi-luxonis @Erol444

themarpe commented 2 years ago

@rsing92 No update yet unfortunately - currently busy on other aspects of the platform

alex-luxonis commented 2 years ago

@rsing92 Do you have an OAK device with microphones, and if yes which one?

rsing92 commented 2 years ago

@rsing92 Do you have an OAK device with microphones, and if yes which one?

I don't have one, but I would love to get one. Check my OG post. Can I run it with data provided from host?

I don't understand, if it's already done, why don't you release it? Will this feature be released? It seems like you spent a lot of time on it, so why not releasing it? Every other node in the depthai pipeline support data provided from host..