0.3 update thread (WIP)

[SirLynix]

Some update about the next obs-kinect version!

General stuff

I massively refactored the code, and improved performance. You can now have multiples Kinect v2 sources at the cost of one (Kinect images/color to depth mapping will only be done once instead of once per source, dramatically improving performance). I'm not sure if that's gonna be helpful but still, it's an improvement.

I also added body and depth hybrids (as requested by someone on reddit):

• "Body or Depth" (everything in the depth range or a body)
• "Body within Depth" (a body inside the depth range)

One of the big stuff about the refactor is also the support for multiple Kinects from multiple frameworks!

This is done by having all code related to a Kinect runtime (or even all Windows specific stuff) in a separate .dll which is loaded by obs-kinect at startup. This was made to add support for freenect(2) in the future, making the plugin available on Linux and macOS (won't be in the 0.3).

There's also a new "Kinect service priority" combo box, which tells obs-kinect to raise the priority of the KinectService.exe process, this is helpful in case where your CPU is under heavy load (like compiling/playing games) and prevents/reduce lost images. (this is only an issue with Kinect runtime 2.0, afaik all other frameworks, including Kinect runtime v1.0 and freenect, don't have a separate service for this).

Also:

• I fixed depth/infrared distortion, changing to depth/infrared source now resize the Kinect source to the right size
• I added a "dirty depth" max count setting, which allows the source to fetch a depth value from a previous frame if it doesn't have a value for some pixel. This helps with flickering but introduces a "depth lag" in case of movement.

About Kinect v1

As you can see, I'm now able to retrieve Kinect v1 color and depth streams. I'm even able to retrieve both of them at the same time (which isn't as straightforward than with Kinect v2) and try to do some green-screen stuff.

... Okay that part still needs some work.

Interesting notes about Kinect v1:

• It's a pain in the ass to use compared to Kinect v2 SDK.
• It has lower color/depth quality than v2 (duh)
• It has an accelerometer and a motor allowing to look up/down on its own.
• It has a "near mode", which brings the depth range from 80-400cm to something like 40-200cm
• It works by exclusive access, which means you can't have obs-kinect and another process using the same kinect at the same time (you can still have multiple obs-kinect v1 sources).
• You can have multiple Kinect v1 at the same time, obs-kinect will ask you which one you want to use (warning: afaik only the first one can have body stream due to a SDK limitation).
• It allows to change white balance/contrast/hue/saturation/exposure/brightness/gain/backlight compensation settings. Kinect v2 doesn't allow that unfortunately (but I'm looking to add support for it through raw USB commands, since the device does support it).
• It has a dedicated background removal stream

That last one was a surprise for me, it also works quite well (even better than my own background removal with Kinect v2, except for the color quality).

Here's an example from Microsoft SDK:

I wish to be able to have something this smooth for Kinect v2 one day.

Anyway, I'll expose this new source for Kinect v1 (unfortunately v2 doesn't have that).

I have no idea when 0.3 will be out, things complicated a lot because of all the Kinect v1 possibilities (and because it's working in such a different way from Kinect v2). I'll update this thread if I have anything relevant to say about this.

I'm sometimes streaming obs-kinect dev on Twitch

I worked on obs-kinect 0.3 body/depth hybrids in english some days ago, you can see the replay here: https://www.twitch.tv/videos/605065358

I also worked on the Kinect v1 support, in my native language (french) here: https://www.twitch.tv/videos/607021382

Enjoy!

[SirLynix]

Body and depth filtering work.

https://twitter.com/SirLynix/status/1256359447947083779

Depth filtering is a lot more unreliable than with Kinect v2 though.

[JonathanFly]

A lot of progress! I'm a bit disappointed in V1 RGB quality but I knew the resolution was low. The background removal is impressive though.

It allows to change white balance/contrast/hue/saturation/exposure/brightness/gain/backlight compensation settings. Kinect v2 doesn't allow that unfortunately (but I'm looking to add support for it through raw USB commands, since the device does support it).

Oh, that would drastically improve general webcam usability, hope you can get that to work.

I'm too lazy to build this myself but if you post a beta build I can try both Kinects this week.

[SirLynix]

A lot of progress! I'm a bit disappointed in V1 RGB quality but I knew the resolution was low. The background removal is impressive though.

I though it was going to be worse than that but yeah, it's outputting a 480p color image (opposed to the 1080p of the v2).

Kinect v1 does support 1280x960 color images at 15 FPS, I'll add support for this as well.

Oh, that would drastically improve general webcam usability, hope you can get that to work.

I think I'll be able to do that on v2 by using a library made by Microsoft (found here), unfortunately it's not open-source and is only compiled for x64, which means I won't be able to do that on the 32bits version of this plugin (shouldn't be a big deal though, I think almost everyone uses a 64bits OS now).

I'm too lazy to build this myself but if you post a beta build I can try both Kinects this week.

I will! 0.3 is really a big update and will require a lot of testing before being out. You can expect a release candidate release in the next few days!

[SirLynix]

Example of 1280x960 output from Kinect:

[SirLynix]

I added support for infrared output. It seems Kinect doesn't like having both infrared and body frames at the same time though (edit: it doesn't like color and ir at the same time neither).

[SirLynix]

I feel stupid.

I managed to halve frametime by... not processing the same image twice. Yup. I forgot to do that.

Fixed in aac42cae1055fbfed46e17ac4071dcf86bef98a4

[SirLynix]

Release candidate binaries are coming soon!

[SirLynix]

Dedicated background removal (using Kinect Developer Toolkit background removal) is now supported for Kinect v1.

This is the best background removal I can provide for Kinect v1, Kinect v2 won't have it (due to Microsoft not releasing a similar tool for it).

I'm now working on the translations keys and then proceeding to release obs-kinect 0.3 RC!

[SirLynix]

First release candidate for v0.3 is out! Please test it and report any bug you encounter by creating an issue.

[SolidSouless]

I just tried this addon with Kinect V1 360 and It perfectly maps me out of the picture and leaves the background, I can't seem to reverse it so the background is removed.

[SirLynix]

Haha, first Time I hear that. Could you please post a screenshot so I get to see what it looks like? Could you also post all the settings you're using?

[SolidSouless]

Well, I just found out it was because I was too close! When I back up a bit more, It works fine. lol Here are the screenshots of what it looked like and the settings I had.

[SirLynix]

Oh right, the Kinect for 360 cannot read depth below 80cm in front of it (up to 400cm), Kinect for Windows can bring this range to 40-200cm by enabling "near mode". Kinect v2 (xbox one) has a range of about 50-500cm.

There's nothing I can do against this, as it's an hardware limitation.

[SolidSouless]

Nah I get it! It was just my own stupidity, It works really well! Props to u my dude 👍

[SirLynix]

Hello guys. It's been a few days since 0.3 RC2 got out and I fixed some minor stuff already for RC3/0.3 release.

Did you encounter some bugs/crashes I could fix before releasing 0.3? Please tell me, I don't want to have to release a 0.3.1. 😄

Also, there's one or two keys missing in the german translation, could someone take a look at that?

[JonathanFly]

I only tried V2 for a few minutes. No bugs or crashes. The main thing I noticed is my green screen error is actually pretty consistent, in my case shifted to the left. If I could nudge the cutout by X percent or pixel that looks like it would dial it in pretty well.

[SirLynix]

The main thing I noticed is my green screen error is actually pretty consistent, in my case shifted to the left. If I could nudge the cutout by X percent or pixel that looks like it would dial it in pretty well.

That's weird. Could you show me what it looks like?

[delet-1]

I'm using a V1 Xbox 360 version and camera options like auto-exposure, brightness, exposure time, frame interval and gain doesn't change anything. Camera elevation does work correctly.

[SirLynix]

Did you try to disable auto-exposure and play with the other options?

I fixed a bug recently which prevented device-specific settings to be applied at device acquisition (when the source shows up, when OBS is started, etc.).

[delet-1]

Yes, I've tried it with different combinations.

[SirLynix]

Okay so you have a Kinect for 360, maybe that's the same issue as #13 and changing camera parameters is not supported on that model. Could you try to use Microsoft Developer Toolkit Browser and run the Kinect Explorer example? It has some options to controls those settings.

If that doesn't work I guess I can't do much. I wonder if libfreenect is able to do this.

Edit: Oh I forgot to ask, do you have errors in OBS log?

[delet-1]

No errors in the logs. I've checked Kinect Explorer and the only options are related to resolution or picture type (rgb, yuv, infrared, rawbayer), no mention of brightness or such.

[SirLynix]

Here, try adjusting exposure settings to see if it's changing something.

[delet-1]

The option is missing so I guess Xbox 360 v1 doesn't support it at all.

[SirLynix]

Oh right. It's a bug in obs-kinect actually, if the Kinect doesn't support those settings the options shouldn't appear. Fixed in 4e2f5f5038afadef00c70dddd19f007ee62de8d4.

It appears to be a limitation of the Microsoft SDK, can't do much about it (except hide the color settings).

[GhostBishop]

I absolutely love this! Finally found a use for my old Kinect V2 lol. But one thing tho is there a way to better compensate for the exposure? Mine is getting blown out on the highlights and I can't seem to keep it under control and obviously the v2 doesn't support brightness and exposure controls. :(

[SirLynix]

Hi and thank you.

There is a way to control exposure by sending raw USB commands. Microsoft did release a lib for this a while ago (only for x64 though), this is the final touch 0.3 is missing to be released (and I gotta say this is taking more time than I though).

Expect the next 0.3 RC (the final one I hope) to have this feature :)

[JonathanFly]

The lack of exposure controls are why I stopped using this for the moment. It looks pretty good in an evenly lit room though.

[SirLynix]

It's been so long, but 0.3 will be available soon, with Azure Kinect (v3) support (see #33) and camera control for v2 I hope!

[SirLynix]

Azure Kinect branch is finally merged (although there are some issues remaining), and I've finally started working on Kinect v2 exposure.

It's going well! I'm using NuiSensorLib released by Microsoft, which allows to send USB commands to the Kinectv2. So far I've been able to retrieve some data I couldn't get with the official SDK (like accelerometer values, which means I have now access to all three Kinect models accelerometer data) and change the LED brightness. I will try to manually set exposure today.

One downside of this is that NuiSensorLib has only been released for 64bits, which means the 32bits obs-kinect plugin won't be able to use it (shouldn't be that much of a problem, everyone really should be using 64bits).

[SirLynix]

Color camera control is coming for KinectV2! However it's a bit hacky for now, as I basically have to reverse engineer the effects of the USB commands I send to the Kinect.

[SirLynix]

Hello guys!

Here's the most recent build of obs-kinect 0.3, or should I say 1.0. It's still a work in progress and I would like some feedback of the recent effort I put in it.

A lot happened since the 0.3 RC 2:

• obs-kinect now supports the Azure Kinect (see #33)
• You can now control the camera settings of all Kinect models*, including the Kinect for XBox One (v2)!
• Instead of removing the background you can now blur it or replace it by an image.
• A lot of little fixes.

*: except Kinect for 360 which doesn't have this capability 😞 (Kinect for Windows has it)

There's also a "dump camera settings" button which can be used to print your current camera configuration to OBS logs, this may be useful when adjusting exposure and such. Initially this was a debug feature but I though it may be interesting for some of you.

You'll notice there are a lot of missing locales, I didn't add them yet for the color settings.

obs-kinect-camera-controls.zip (copy the obs-studio folder onto C:\Program Files\obs-studio).

Before releasing this as a release candidate, could you guys test it a bit, in order to see if some issues are remaining? That would help me a lot!

The lack of exposure controls are why I stopped using this for the moment. It looks pretty good in an evenly lit room though.

It's there now! @JonathanFly

I absolutely love this! Finally found a use for my old Kinect V2 lol. But one thing tho is there a way to better compensate for the exposure? Mine is getting blown out on the highlights and I can't seem to keep it under control and obviously the v2 doesn't support brightness and exposure controls. :(

It does now! @PaceDesigns

[Milliw]

I copied over the files from your zip file but as far as I can see, not much changed on the interface.

I do have an option "KinectSDK background removal" but it is greyed out. Don´t know if it was there before. (Kinect v2) The blur background effect is not available. I compared your images and it´s different to my gui.

Wanted to translate the "; obs-kinect-azuresdk backend" stuff but as I do not see what it does I´m pretty unsure of the terms to use for German translation.

[SirLynix]

I copied over the files from your zip file but as far as I can see, not much changed on the interface. The blur background effect is not available. I compared your images and it´s different to my gui.

Are you running obs-studio in 32bits mode? The last archive only has the 64bits dll.

[Milliw]

My fault. Something did not work well while copyying the files over from the archive. Like only a few files were copied. Checked the files/folders manually and now it´s working.

How to translate? I went into the locale file on Github but that´s not the most recent. Shall I manually edit the local file and send it to you?

[SirLynix]

Glad to hear it's working. Did you try the color settings?

The most recent locale file is on GitHub here but it's missing a few keys, give me about 20minutes and I'll add them 😄

[SirLynix]

Took a bit more than 20min but here you are, this build is updated with the last locales files. I put a "; missing translation" at the end of keys missing a translation in de-DE.ini and pl-PL.ini, but you can check the others if you think you can improve on it! obs-kinect-locales.zip

[Milliw]

Glad to hear it's working. Did you try the color settings?

I tried. Works so far. But OBS crashes repeatingly when switching background effect modes. Got several crash logs.

Translated most of the missing strings. Some I´m unsure about and have commented them. Made a pull request. Hope this is the right way?! ^^

Crash 2020-12-26 15-55-36.txt Crash 2020-12-26 15-50-43.txt Crash 2020-12-26 15-47-56.txt

[SirLynix]

Oh, that's weird. So you can't enable blur effect? I don't have this issue.

And yes, the PR is the right way to do!

[Milliw]

Oh, that's weird. So you can't enable blur effect? That's weird, I don't have this issue.

No, I can. But it crashes sometimes when switching the mode. Maybe it has to do with the translation and german special characters?? Just an idea...

[SirLynix]

It looks like a rendering related crash, what's your GPU?

[Milliw]

RTX 2060

[SirLynix]

Maybe try updating your drivers, I'm having a RTX 2080 Ti and no issue. The logs seem to be pointing a driver crash. It would be easier if I could reproduce any of theses crash myself.

[Milliw]

I´m on 460.79. Should be quite recent. Wanted to update/look up for the newest driver but I can´t complete the form on the nvidia page. Tried three different browsers... guess nvidia doesn´t like me

[Milliw]

Updated to last 460.89 WHQL by downloading it from third party site. Still same issue. Works when I first open the settings and crashes then when changing the background mode several times. Not a big issue maybe. I can work around.

Great update btw! ;)

[SirLynix]

Thanks! All I need to do now before releasing this as the 1.0 version is to make it work with Linux using libfreenect. Shouldn't be that complex 😄

[Milliw]

One more note: The cameras I have in OBS from earlier version of your plugin don´t work out of the box. Only after removing/rechecking the greenscreen imitation checkmark it does.

Removed all old Kinect sources. Added a new one. Changed background effect -> crash. Thought removing old ones could maybe solve the issue.

[Milliw]

Unticking the checkbox to use gpu to compute color-to-depth-calculation leads to some weird background removal. The cut area stays the same as before and not adapting to movement anymore. (Wanted to try to let the cpu do the workload as my gpu is not the strongest)

(Kinectservice.exe uses up to 14% and obs up to 12% gpu load according to task manager. Without streaming. Quite heavy load in my oppinion.)

[SirLynix]

I'll take a look at that, thanks for letting me know

[fotiDim]

@SirLynix congrats on the great work! After testing the build you posted above I spotted the folllowing issues:

• Crashes (sometimes) when switching the background mode (etc. blur). I am attaching some crash logs
• I am not sure how manual white balance mode works. I would like to be able to set the light temperature in Kelvin.
• I am not sure what the "Unknown" white balalance mode does.
• Most drop downs have values with key names (e.g. ObsKinectV2.WhiteBalanceMode_Auto)
• Switching between 50hz and 60 hz mode does not seem to have any effect. I do see some flickering on the camera feed.

Crash 2020-12-27 23-44-34.txt Crash 2020-12-27 23-34-55.txt Crash 2020-12-27 23-35-29.txt