search

SirLynix / obs-kinect

OBS Plugin to use a Kinect (all models supported) in OBS (and setup a virtual green screen based on depth and/or body detection).
GNU General Public License v2.0
433 stars 30 forks source link

Azure Kinect support #33

Closed SirLynix closed 3 years ago

SirLynix commented 3 years ago

Good news everyone.

Microsoft contacted me about this plugin and is willing to help me support the Azure Kinect by sending one to me so I can add support for it!

This also means I will work on the plugin again, after not touching it for a few months due to other projects.

I'll use this thread to keep you informed about azure kinect support in obs-kinect.

SirLynix commented 3 years ago

Just received the beast! image

cefoot commented 3 years ago

I'm looking forward to this! If I can assist you (dev or test) let me know!

SirLynix commented 3 years ago

Initial Azure Kinect support is done (color, depth, infrared and depth to color mapping which means the greenscreen works).

image

I had to put a hat on because Azure Kinect has some troubles with my hair, but it flickers a lot less than v2. Body tracking support is on its way but it's gonna be a little bit more complex than with Kinect v1/v2 because it's a separate SDK/lib.

Shouldn't take long

SirLynix commented 3 years ago

image

Shouldn't take long before a test version. Since I never properly released 0.3, this will be part of this release!

cefoot commented 3 years ago

Hi looking really forward to this.. so much that I tried to build it after I added some missing parts to the conflig.lua I was able to build unfortunately it craches when I try to add the kinect Source to a scene. But since this is my first try with building an obs plugin the problem is probably on my side ;)

SirLynix commented 3 years ago

You're probably missing a few plugins files, but don't worry, packages are coming very soon!

SirLynix commented 3 years ago

Here's a test package of obs-kinect with Azure Kinect support (64bits only for now): obs-kinect-azure-test-fixed.zip

Unpack the obs-studio folder to your obs-studio folder. Please note this version doesn't include body filtering files for Azure Kinect! It will try to load them at runtime but will disable body filtering if it fails.

I've updated the README for Azure Kinect if you want to test body filtering.

Please let me know how it works (or not) for you, and let me know of any bug you encounter!

amourao commented 3 years ago

@SirLynix Thank you for taking the time and adding support the Azure Kinect so quickly.

I have an additional request and offer to help: as you may know, the Azure SDK is also available for Linux (tested on Ubuntu 18.04). How hard would it be to try and compile it on Linux?

This is also sort of related to SirLynix/obs-kinect#9

SirLynix commented 3 years ago

Actually I planned already to release the next version of obs-kinect on Linux (with Azure Kinect support only for now, I hope to bring Kinect V1 and Kinect V2 support next), don't worry!

It shouldn't be difficult, I've been thinking of supporting Linux (and macOS) since I started working on 0.3, at this point I think the code is close to ready (it doesn't rely on Windows specific code, which only exists in Sdk10 and Sdk20 backends).
I'll give it a try soon!

cefoot commented 3 years ago

I changed the folder in the zip from obs-studio\bin\64bits to obs-studio\bin\64bit (without s) and everything is working fine! :) kinect_v2

just a short first test for now ;)

SirLynix commented 3 years ago

You're right, I messed up the folder name. I uploaded a fixed package in my original message.

Glad to see it's working! It looks great! Did you try body tracking yet? you'll have to install Azure Kinect Body Tracking Sdk dependencies for that.

And some new cool effects are coming as well #35

Funkcorner commented 3 years ago

Does the Azure Kinect cause problems with its very high resolution video? 12MP - that's a lot more for the GPU to think about...

SirLynix commented 3 years ago

4K video is pretty fine on my computer, even with green screen enabled (taking about 20% of GPU). Except for some lag (probably due to depth to color mapping which is pretty compute-heavy especially on 4K).

reneschulte commented 3 years ago

Awesome work! Gave it a try and it works really well, also Body masking, though I disable it as it's too heavy on my GPU. I noticed the typical holes that can appear with depth data. Are you thinking about some extra image processing to mitigate that like dilation and erosion filter combo?

SirLynix commented 3 years ago

Hello, and thank you!

Yes I'm thinking of using post-processing to improve the filtering. I'm also trying to see if that can be improved at the Azure Kinect level (see here).

reneschulte commented 3 years ago

Makes a ton of sense @SirLynix that it should be improved by the AK itself. We both have the same problem with hair color! 😁 https://youtu.be/WhT3ZpQrvf0

Great work and glad Microsoft donated that Azure Kinect and you got it quickly implemted. 😉

SirLynix commented 3 years ago

Thanks a lot for the video! Glad my plugin is useful to you!

I have a lot of work to do for other projects right now but I will get back to it soon and try to release the (hopefully) stable version asap!

dotMorten commented 3 years ago

Does this support more than one kinect camera, like the "official" green screen example? https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen#why-would-more-than-1-camera-be-used

SirLynix commented 3 years ago

Does this support more than one kinect camera, like the "official" green screen example? https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen#why-would-more-than-1-camera-be-used

It does support multiple Kinect when using multiple sources but doesn't support combining multiple Azure Kinect into one result yet. However that's something I'm thinking about, it could be cool to support it (I won't be able to test it myself however).

dotMorten commented 3 years ago

Time for Microsoft to send you a second one :-)

It would help with the shadow effects I see here: image

SirLynix commented 3 years ago

It would help with the shadow effects I see here:

Yes indeed, but I will soon try to work on a post-process solution (see #32 )

cefoot commented 3 years ago

this could also help I guess https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen

SirLynix commented 3 years ago

this could also help I guess https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/examples/green_screen

Yes I did check the code. Nothing too complex since it relies on a computer vision library to get the transformation between the two cameras. I think I can do something similar but I'll need someone with two azure kinect to test it.

cefoot commented 3 years ago

thx again! I had so much fun in a meeting today! https://youtu.be/PgZNRoZY8rM

SirLynix commented 3 years ago

The latest build can be found at the end of #7 with color camera settings for Azure Kinect!

Ahmed-KASAPBASI commented 2 years ago

I need to record videos (RGB, depth and skeleton) with voice at the same time (synchronously) from 3 different Azure kinect which are connected to one PC.

How I can do that? can OBS do that, what are the possible formats of the recorded videos ? Please, can you help me?

SirLynix commented 2 years ago

Hi.

I doubt obs and obs-kinect can really help you, you should try using the Kinect SDK directly: https://github.com/microsoft/Azure-Kinect-Sensor-SDK

Ahmed-KASAPBASI commented 2 years ago

Hi.

I doubt obs and obs-kinect can really help you, you should try using the Kinect SDK directly: https://github.com/microsoft/Azure-Kinect-Sensor-SDK

Thank you so much for your reply. I searched a lot about this topic but I could not get an answer. When I searched, I found that Azure kinect SDK can not do that. Do you have an other advice, please. OR do you know any person can help me?

SirLynix commented 2 years ago

Of course the SDK can do that, it has its own recorder: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/tools/k4arecorder

Ahmed-KASAPBASI commented 2 years ago

Of course the SDK can do that, it has its own recorder: https://github.com/microsoft/Azure-Kinect-Sensor-SDK/tree/develop/tools/k4arecorder

Thank you a gain. Yes, you can record RGB videos without the voice but in my issue I need to record 3 types of videos (RGB, depth and Skeleton) with voice at the same time.

SirLynix commented 2 years ago

I understand, I'm not saying k4arecorder is the solution to all your problems, but it shows how you can record color and depth using the SDK, adding skeleton and voice shouldn't be a big deal (see k4aviewer which records audio).

For skeleton tracking, see https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download

See also other examples for synchronisation between Azure Kinect.

I don't think there already is a tool which does everything you need, you'll have to code it yourself, feel free to look how I use Azure Kinect in obs-kinect if that helps)

Ahmed-KASAPBASI commented 2 years ago

I understand, I'm not saying k4arecorder is the solution to all your problems, but it shows how you can record color and depth using the SDK, adding skeleton and voice shouldn't be a big deal (see k4aviewer which records audio).

For skeleton tracking, see https://docs.microsoft.com/en-us/azure/kinect-dk/body-sdk-download

See also other examples for synchronisation between Azure Kinect.

I don't think there already is a tool which does everything you need, you'll have to code it yourself, feel free to look how I use Azure Kinect in obs-kinect if that helps)

I appreciate your patience. Final question, do you know any program or software that can help me to get Skeleton videos from RGB videos? If there is, I can solve the Skeleton videos problem instead of recording the Skeleton videos separately.