search

microsoft / MixedReality-WebRTC

MixedReality-WebRTC is a collection of components to help mixed reality app developers integrate audio and video real-time communication into their application and improve their collaborative experience
https://microsoft.github.io/MixedReality-WebRTC/
MIT License
903 stars 278 forks source link

can MixedReality-WebRTC support Azure Kinect DK? #140

Open zhuangce opened 4 years ago

zhuangce commented 4 years ago

my project work fine on hololens ,i try to use Azure Kinect DK as my pc's camera ,but when i press "play " buttun in unity ,it just crashed . since unity can recognize a kinect device as a normal camera , i 'm wondering if it's because webrtc dont support Azure Kinect DK?

djee-ms commented 4 years ago

This is a bug in Google's code, they don't support 7 microphones and crash. See for example https://bugs.chromium.org/p/webrtc/issues/detail?id=10881. There are multiple issues reported for that, and as far as I know it is not fixed yet. Also Azure Kinect DK was never fully tested with MixedReality-WebRTC since I cannot get past this crash myself.

zhuangce commented 4 years ago

@djee-ms i see. Thanks for your reply

chrisse27 commented 4 years ago

@djee-ms Does the patch that you are mentioning in #141 (https://webrtc.googlesource.com/src/+/eb8c4ca608486add9800f6bfb7a8ba3cf23e738e) fix this issue as well?

chrisse27 commented 4 years ago

Btw. you can disable the audio device of the Azure Kinect in the device manager of Windows to avoid the crash.

djee-ms commented 4 years ago

See https://bugs.chromium.org/p/webrtc/issues/detail?id=10881 for a more complete picture. If the question is about avoiding the crash, but without using the device, then yes it's probably enough. If the question is about actually using the 7-channel audio of Azure Kinect DK, then the patch is necessary, but I didn't try it to confirm it's sufficient, and the WebRTC devs seem to suggest 7-channel audio won't work anyway, only mono/stereo are supported, so opening the Azure Kinect DK in stereo is the only realistic way to go as of today.

Andrew-Dupuis commented 4 years ago

I'm working on a WebRTC implementation with the Azure Kinect as well - is there a graceful failure we could add (hardware ID filter before the webRTC library call, or at least a try/catch to stop the process from failing)? I'm happy to handle Kinect inputs myself, but I've now run into an issue twice where I have a kinect and a local webcam I'd like to use, and being able to still use the as-built video source code would be nice! The unhandled process crash (especially in Unity) is blocking. As an aside, the Azure Kinect and it's mic module do work with WebRTC on Chromium on Windows, at least in the official API samples repos (https://webrtc.github.io/samples/)