Depth engine doesn't like black hair and black headset a lot.

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.

https://Azure.com/Kinect

MIT License

1.49k stars 619 forks source link

Depth engine doesn't like black hair and black headset a lot. #1394

Open SirLynix opened 3 years ago

SirLynix commented 3 years ago

Describe the bug When using Azure Kinect with a depth output, the reported depth for my black hair and headset is zero.

To Reproduce Wear a headset or have black hair, open Azure Kinect Viewer and look at the depth values (or just see the black area) on the depth image.

Expected behavior I expected Azure Kinect to be able to compute depth in this case, as Kinect for 360 and Kinect for Xbox One don't have this problem. (also the reported infrared values are not zero but ~80 for my hair and ~40 for my headset, I suspect a threshold).

Screenshots

Desktop (please complete the following information):

OS with Version: Using Windows 10 2004
SDK Version: Azure Kinect SDK v1.4.1

Firmware version:

Current Firmware Versions:
RGB camera firmware:      1.6.110
Depth camera firmware:    1.6.79
Depth config file:        6109.7
Audio firmware:           1.6.14
Build Config:             Production
Certificate Type:         Microsoft

nachoreimat commented 3 years ago

yes, I also noticed missing depth on black/dark objects. Is it a problem of the TOF cameras? maybe those objets/materials do not reflect the laser

TomHVR commented 3 years ago

Maybe there are hairsprays that are more reflective of IR light?

Might be worth taking a piece of glass that is IR transparent, and then trying different hairspray products on them to see which has the most IR reflectivity.

edit: we also use this in film/video production as a temp glare reduction on objects. Wipes off easily, Not sure if I'd want it on my hair though! https://www.amazon.com/Krylon-K01310-11-Ounce-Dulling-Spray/dp/B00009R8O6?th=1

qm13 commented 3 years ago

This is an issue with IR based depth sensors in general. If the material either absorbs or scatters the IR light emitted from the sensor it is not going to see the object. We have heard from a number of customers working in the volumetric film making space that hairspray can address the issue. We have not tested it. Please test it on sample fabric before spraying someone's $200 designer jeans with hairspray.

SirLynix commented 3 years ago

I'm sorry but that doesn't help me, as I'm not going to spray my hair everytime I want to use that camera.

I own all three Kinect models and the Azure is the only one to have this issue. As the reported infrared value is not zero I suppose there's a threshold that discards it.

Could there by any way to let the user adjust that threshold?

madhan001 commented 3 years ago

I too have problems with the depth camera not seeing black objects, the kinect v2 I own can perfectly see the object though

SirLynix commented 3 years ago

Please @qm13 consider reopening this issue, as it's not resolved.

qm13 commented 3 years ago

This is a matter of physics. Kinect v2 project considerably more light (hence the large power brick) into the operating space than Azure Kinect. We interviewed a number of existing Kinect v2 customers when designing Azure Kinect. One request was smaller or no power brick (achieved) and willingness to this against the operating space. Azure Kinect sensor is more sensitive than Kinect v2 helping limit the reduction in operating space.

There is currently no way to adjust the threshold. I have passed the request to the team working on an improved depth engine.

TomHVR commented 3 years ago

is there potential for introducing a hole filling algorithm?