[Remote] depth_engine_start_helper reports error 204 ... GPU API failure ... at start of streaming

[StevenButner]

I have a 2 synchronized Azure Kinects and a fully working test setup. When I attempt to integrate the Kinects (on the same hardware system where my test works) within a larger real-time application, the cameras consistently fail at the moment when k4a_device_start_cameras() is called. I get an error message on stdout as below:

[2019-10-01 11:49:11.734] [error] [t=4190] /home/vsts/work/1/s/extern/Azure-Kinect-Sensor-SDK/src/dewrapper/dewrapper.c (154): depth_engine_start_helper(). Depth engine create and initialize failed with error code: 204.

The wrapper layer that handles all calls to/from the k4a library routines is the same as the one I use in my test program .... and the test program works perfectly. It seems to me that this issue must be resource-related or possibly a bug.

My host system is an Intel Core-i7 with 16 Gbyte RAM running Ubuntu 18.04LTS The SDK is version 1.2
Kinect device firmware is as below: == Azure Kinect DK Firmware Tool == Device Serial Number: 000261592512 Current Firmware Versions: RGB camera firmware: 1.6.102 Depth camera firmware: 1.6.75 Depth config file: 6109.7 Audio firmware: 1.6.14 Build Config: Production Certificate Type: Microsoft

Device Serial Number: 000441592912 Current Firmware Versions: RGB camera firmware: 1.6.102 Depth camera firmware: 1.6.75 Depth config file: 6109.7 Audio firmware: 1.6.14 Build Config: Production Certificate Type: Microsoft

I don't know if this is useful, but I tried running the program "deversion" from the SDK and it produced the following report: ~/Azure-Kinect-Sensor-SDK/src$ ../build/bin/deversion [2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.0.0 with error: libdepthengine.so.0.0: cannot open shared object file: No such file or directory [2019-10-01 12:59:13.968] [error] [t=5362] ../src/dynlib/dynlib_linux.c (86): dynlib_create(). Failed to load shared object libdepthengine.so.1.0 with error: libdepthengine.so.1.0: cannot open shared object file: No such file or directory Found Depth Engine Pluging Version: 2.0.0 (current)

Any suggestions for getting past this issue would be greatly appreciated.

   Steve B

[christian-rauch]

There are different uses of OpenGL in the SDK.

Depth Engine + OpenGL for computation:

The Depth Engine converts the raw sensor data from the camera to depth. I guess that it uses OpenGL shaders for this and the OpenGL context is created via GLX, which requires an X11 connection. With EGL in place of GLX, the Depth Engine could run on headless systems without an X11 connection. OpenGL is here only used for computation, not for visualisation/graphics. Once you recorded the depth, there is no use for the Depth Engine anymore.

OpenGL for visualisation:

If you want to visualise the 3D data in the Azure Kinect Viewer you will need OpenGL again for visualising the 3D points. You do not need the Depth Engine for that, but you will still need OpenGL with a connection to a display server (X11, Wayland). Once you moved the log files to a workstation, you can use the Azure Kinect Viewer there to look at the point clouds.

Instead of using the Azure Kinect Viewer, you can of course just compute the projection from depth to 3D yourself, using the camera intrinsic stored in the mkv file. Then you can export point clouds in any format, read them with e.g. numpy and just "render" the point via software. One suggestion to do this is using the ROS node and logging the point clouds into a bag file directly.

[trioKun]

$ export DISPLAY=:0
$ xhost +

Above cmd lines works for me to open k4a camera in ssh sessison.

[lukexi]

@lukexi we are actively working on an EGL version of the depth engine. This will be available in the next release of the Sensor SDK.

Hi @qm13, is there any news on the EGL version of the depth engine to allow the Azure Kinect to be used without X11 since April 2021? Much appreciated!

[rajkundu]

Without the SDK's Depth Engine, while I can easily extract the raw 2D images from the MKV files (and have done so already), they are of little use to me. My understanding is that the proprietary Depth Engine is required to convert/reconstruct this data into 3D. Is this not correct?

Just wanted to make an update in case anyone else finds themselves in the same situation. The most convenient way that we found to compute the 3D point cloud from the 2D depth image (or transformed depth image) without using the SDK was to simply use the fastpointcloud example and its precomputed XY table. This page has some more information on converting 2D depth images to 3D point clouds and is how we found the fastpointcloud example.

In Python, using pyk4a, our code looks like:

def compute_xy_table(calibration: pyk4a.Calibration, shape: tuple, calibration_type: pyk4a.CalibrationType):
    # Compute XY table
    height, width = shape
    xy_table_data = np.zeros((*shape, 2), dtype=np.float32)
    for v in range(height):
        for u in range(width):
            try:
                x, y, _ = pyk4a.Calibration.convert_2d_to_3d(calibration, (u,v), 1.0, calibration_type, calibration_type)
            except ValueError:
                x, y = np.nan, np.nan # can also be 0
            xy_table_data[v,u] = x,y
    return xy_table_data

xy_table_depth = compute_xy_table(calib, depth_img.shape, pyk4a.CalibrationType.DEPTH)
xy_table_color = compute_xy_table(calib, transformed_depth_img.shape, pyk4a.CalibrationType.COLOR)

def compute_point_cloud(depth_image, xy_table):
    return depth_image[:, :, np.newaxis] * np.dstack((xy_table, np.where(np.logical_or(np.isnan(xy_table[:,:,0]), np.isnan(xy_table[:,:,1])), np.nan, 1.0)))

depth_pc = compute_point_cloud(depth_img, xy_table_depth)
color_pc = compute_point_cloud(transformed_depth_img, xy_table_color)

When we compare this computed point cloud to the SDK's, we get a maximum error of 0.500 mm, which makes sense because the SDK rounds to uint16 whereas we use float32 in this case.

As explained by the Azure Kinect SDK documentation, the XY table remains constant for each camera calibration (so, if you never re-calibrate your camera after the factory, then it is forever constant). So, you only have to compute it once, and then, you can load it from disk to compute point clouds on-the-fly. You can even GPU-accelerate the process, e.g. using cupy, to make it extremely fast. We found that with GPU acceleration, we were able to both load frame data/XY table from disk and compute the point cloud (~10 ms on our system) faster than we were able to directly read the 3D point cloud from disk (e.g., saved as PCD file, NPZ archive; ~30 ms on our system). Therefore, computing the point cloud on the GPU using the XY table is both more space-efficient and time-efficient than directly storing 3D point clouds on disk.