search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.49k stars 617 forks source link

Generate a Color to Depth mapping image? #1393

Open SirLynix opened 3 years ago

SirLynix commented 3 years ago

Hi, obs-kinect dev here.

When I made the plugin with KinectV1 (Kinect for 360) and KinectV2 (Kinect for Xbox One) I could generate color, depth and infrared images and a color to depth image (an image containing X and Y coordinates to the depth image for every coordinate).

This was generated using INuiCoordinateMapper::MapColorFrameToDepthFrame for KinectV1 and ICoordinateMapper::MapColorFrameToDepthSpace for KinectV2.

With Azure Kinect, I searched for a similar function but only found functions to convert image from one camera to another (I'm using k4a_transformation_depth_image_to_color_camera).
The "problem" here is that I had to handle this in a different way than I do with KinectV1 and KinectV2, which forces me to set a special case for the Azure Kinect and gives me a bit of trouble (for example, I currently cannot show a body-filtered greenscreen of someone using the color stream and having another source showing a greenscreen of that person using the depth stream, because I would map the body image to the color space with the first source).

All this to ask, is there a way to efficiently generate a color to depth mapping image (containing for every color pixel both XY coordinates to the depth map) using k4a API? Thanks.

qm13 commented 3 years ago

@SirLynix there is a color to depth transformation function k4a_transformation_color_image_to_depth_camera. This is an expensive function (as were its v1 and v2 predecessors) as it first must cast a ray from depth to color and then transform pixels back to depth,

SirLynix commented 3 years ago

Thanks, but as far as I understood it doesn't generate the same thing. Kinect v1 and Kinect v2 generate a color to depth mapping texture (containing coordinates). If I understood correctly this function directly maps the image.

JamesMcCullochDickens commented 2 years ago

@qm13 I am trying to write a python equivalent of this API call and am having difficulty finding the source code for the k4a_transformation_color_image_to_depth_camera function, do you have a link? Thanks.

qm13 commented 2 years ago

@JamesMcCullochDickens this is a request (enhancement) to add this function to the Sensor SDK. The function does not exist today.

JamesMcCullochDickens commented 2 years ago

@qm I guess I'm not sure what you mean, the function k4a_transformation_color_image_to_depth_camera doesn't exist? It's part of the docs. I just mean if I could see the c++ source code and then I can study it and write a python equivalent, not a wrapper, but with numpy etc. Thanks for the help.

qm13 commented 2 years ago

Sorry @JamesMcCullochDickens confused your request with the enhancement request of this issue.

You can find the C source for k4a_transformation_color_image_to_depth_camera in this SDK source file.