microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.5k stars 619 forks source link

Python API doesn't transform correctly to point cloud from depth image #1556

Closed UnaNancyOwen closed 3 years ago

UnaNancyOwen commented 3 years ago

Describe the bug

Python API doesn't transform correctly to point cloud from depth image. The API returns a result, but it does not seem to be the correct point cloud.

To Reproduce

  1. run this simple script for save point cloud to file

    import numpy as np
    import k4a
    
    if __name__ == '__main__':
        # Open Device
        device = k4a.Device.open()
    
        # Start Cameras
        device_config = k4a.DEVICE_CONFIG_BGRA32_1080P_NFOV_UNBINNED_FPS15
        device.start_cameras(device_config)
    
        # Get Calibration
        calibration = device.get_calibration(
            depth_mode=device_config.depth_mode,
            color_resolution=device_config.color_resolution)
    
        # Create Transformation
        transformation = k4a.Transformation(calibration)
    
        # Capture One Frame
        capture = device.get_capture(-1)
    
        # Get Point Cloud
        xyz_image = transformation.depth_image_to_point_cloud(capture.depth, k4a.ECalibrationType.DEPTH)
    
        # Save Point Cloud To Ascii Format File
        height, width, channels = xyz_image.data.shape
        xyz_data = xyz_image.data.reshape(height * width, channels) # convert to (1080 * 1920, 3) from (1080, 1920, 3)
        np.savetxt('data.txt', xyz_data, delimiter=' ', fmt='%u') # save ascii format (x y z\n x y z\n x y z\n ...)
    
        # Stop Cameras
        device.stop_cameras()
  2. open ascii format point cloud data using meshlab 2021-04-02_16h01_18

Expected behavior

It can get correctly transformed point cloud data.

Logs

Screenshots

Desktop (please complete the following information):

Additional context

The other features of Python API are working correctly.

UnaNancyOwen commented 3 years ago

@JonathanESantos What do you think? Do you have any advice?

JonathanESantos commented 3 years ago

There is a bug in the transformation function depth_image_to_point_cloud() where a ctypes.c_uint16 is used. It should be ctypes.c_int16. The X and Y coordinates are centered around 0 and have negative numbers.

The docstring is also a bit misleading where it says each pixel is a set of (x, y, z) values, which sounds like interleaving. In fact, the output images are arranged in planes, where the first plane is X, the next plane is Y, and the last plane is Z.

The PR to fix this is 1568. The example code to output the point cloud in a text file has also been added for reference.