static scene masking trick

[BobTape]

Hi SirLynx, i will try to make new masking layer to your great Kinect plugin. I think in scene are a lot of static object, that we need to preserve in scene, but they are sooo shiny and depth sensor have problem with depth detection on it (for example turnables with vinyl on it).

My idea is add to plugin new static layer with optional mask (some black and white mask (maybee with grayscale for smooth)) this layer have high priority above depth sensor mask. (in first version i planing this layer you can make from any tool like a gimp from scene screenshot).

Result: static things in scene (microphone, desk and my dj stuff will always visible), because currently is on it a lot of depth sensor noise. This dramaticaly improves faux green screen efect on static scene, when is this type of masking needed.

Thanx for reply your ideas about it. I can help you with code, (after i learn how it works, because im not typical c++ guy, i currently work most only with c# and typescript).

Bob

[Milliw]

Interesting! :-) -- Gesendet von /e/ Mail.

Am 2. April 2021 12:24:03 MESZ schrieb Bob @.***>:

before: after:

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/SirLynix/obs-kinect/issues/52#issuecomment-812471352

[BobTape]

And some example before: After:

this is only example (not functional implementation), but here is for example my shiny mobile phone invisible. yes it have some cons (some glitches on the edge between mask and kinect deepth sensor mask) i now :D

[SirLynix]

Hi.

This idea sounds nice, so if I understand correctly you're suggesting a "passthrough" mask (not sure what it would be called) that would take precedence over the depth filtering, in order to allow to let some parts of the scene visible.

How would that work? Full white = always visible and full back = depth filtering (default behavior), with grey being at least semi-visible (and fully visible if depth filtering passes)? I can also think of something a bit more complex/powerful: full white = always visible, full black = never visible with alpha controlling the opacity of the mask (opaque being always visible/invisible and transparent being depth filtered).

Also, if I understand correctly, this is already possible using multiples source in OBS (one RGB source with mask and one faux green screen underneath or something like that). Note that I would still implement it as it seems useful (and pretty simple to do).

[BobTape]

Yes exactly you understood that well as me. "passthrough" mask sounds logicaly. I think, when make it white (visible by depth)/transparent scale (semi visible) /black (invisible) can be ok, if its simple to implement it.

Yes this trick can be handled with multiple kinect source, but is not comfortable when you scale layers (must be well aligned). From my angle of wiew this can be simple to implement it, but im not good at c++ currently :D (i forget some skills)

Thank you SirLynix

[BobTape]

This week i finaly got USB3.0 pci-express card with Renesas/NEC UPD720202 chipset (propably one of several functional chipset for stable 30 FPS on kinect sensors on Windows 10). And i decide this aproach as usable :)

SirLynix is it from your angle of view is this ok ? What you use for good FPS results ?

[SirLynix]

As long as you manage to get a stable 30 FPS I'd say it's good. My Kinect for XBox One is plugged on my motherboard directly (it's a MSI MAG B550 TOMAHAWK), never had much troubles with it. However my previous computer (which had a MSI Z97-GD65) had some USB issues.

[BobTape]

Thank you. I dont know how solve this yellow exclamation marks in Configuration viewer, but i think now its functional. (propably more people have this state).

[BobTape]

Hi sir linx, what you think about this https://github.com/microsoft/ailab/tree/master/VirtualStage Microsoft use this https://grail.cs.washington.edu/projects/background-matting/ for making virtual green screen postproduction software. Maybee this technoogy can be integrated to this plugin :) Their make 5 sec of blank scene screen for help AI to learn what is needed to masking. Nice idea.

[SirLynix]

I've found https://github.com/PeterL1n/BackgroundMattingV2 which is the evolution is this. It sounds really great, however I don't think it's related to obs-kinect since it doesn't require a Kinect (maybe I'm wrong but I have the feeling it doesn't benefit from the depth sensor). AI is also currently outside of my competence field.

I think BackgroundMatting belongs to another OBS plugin (since it doesn't require a Kinect), made by someone who knows how to use tensorflow.

[SirLynix]

@BobTape Sorry I took so long to implement this, but here you are, now available in obs-kinect last builds.

The visibility mask works like this: white means always visible, black means never visible and fully transparent means default behavior (and of course you can use any color/alpha between these).

Could you test it and let me know if it works as expected? (it's the green screen "visibility mask" option)

[BobTape]

Hi SirLynix i try it soon as posible. Thanx a lot for this. :)

[BobTape]

Hi Sir Lynx, Nice i like it, it work very well :) you are hero. I will test it.