Color Image Capture Fails on Second Request

[constantin-ulbrich]

Hi Kinect DK community,

I have been trying to use the Kinect for a computer vision application in my university thesis project. As I haven't used the Kinect before and only have basic coding knowledge in C, I thought that I familiarize myself with the Kinect by building a simple console application first. This application uses to a large extent the example code provided in the Microsoft Docs (https://docs.microsoft.com/en-us/azure/kinect-dk/retrieve-images). The application should capture an image and then retrieve the Color-, IR-, and Depth-Image. Subsequently, for each of the retrieved image types the application should print to the console the resolution and stride of the image if the application retrieves the images correctly. If the retrieval of the image is unsuccessful, then the message 'Image is null!' should be printed to the console. The problem arises if I run the code more than once, after plugging in the device. When running the code the second time, the application fails to retrieve the Color-Image from the capture. Unfortunately, being able to retrieve the Color-Image plays a central role in the application I'm building for my thesis. Sadly, after several hours of debugging and attempting different solutions, I still could not fix this bug. Also, I am starting to be a little clueless, what the issue could be. Has anyone experienced a similar issue or is able to help?

Some additional information:

• I am working on a windows machine (running on Windows 10)
• To compile and run the code, I am using Visual Studio Community Edition (2019) v16.6.3
• I have installed the Microsoft Azure Kinect Sensor 1.4.1 package, using the Nuget package manager
• I have also the Azure Kinect Viewer v1.4.1 installed and when opening the Kinect camera with it, it displays all image types perfectly fine

Here are the steps to reproduce the issue:

1. Plug in the device.
2. Run the application code. Then you get the expected outcome.
3. Run the application again. Then you get no color image returned, but the depth and color image still work.
4. Disconnect device and reconnect it.
5. Run the application code. Then you get the expected outcome.
6. Run the application again. Then you get no color image returned, but the depth and color image still work.

Is there something which is missing to initialize the device or something which needs to be done to properly close the device?

Below, you can find the output of a successful run, the output of an unsuccessful run, and the source code.

Thank you very much for your help!

Output after first time running the code:

We are in the azure_kinect_controller code.
Kinect is switched on!
Camera Started
Capture Success
 | Depth16 res: 720x1280 stride:    0
 | Depth16 res: 576x 640 stride: 1280
 | Depth16 res: 576x 640 stride: 1280

Output after running the code the second time (note, there are only two image resolutions, as well as the message 'Image is null!' printed):

We are in the azure_kinect_controller code.
Kinect is switched on!
Camera Started
Capture Success
Image is null!
 | Depth16 res: 576x 640 stride: 1280
 | Depth16 res: 576x 640 stride: 1280

Source Code:

#pragma comment(lib, "k4a.lib")
#include <k4a/k4a.h>

#include <stdio.h>
#include <stdlib.h>

int main() {
    int returnCode = 1;
    int TIMEOUT_IN_MS = 5000;
    printf("We are in the azure_kinect_controller script.\n");
    k4a_device_t device = NULL;
    k4a_capture_t capture = NULL;

    k4a_device_configuration_t config = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
    config.camera_fps = K4A_FRAMES_PER_SECOND_30;
    config.color_format = K4A_IMAGE_FORMAT_COLOR_MJPG;
    //config.color_format = K4A_IMAGE_FORMAT_COLOR_BGRA32;
    config.color_resolution = K4A_COLOR_RESOLUTION_720P;
    config.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED;

    // Open the first plugged in Kinect device
    if (K4A_FAILED(k4a_device_open(K4A_DEVICE_DEFAULT, &device)))
    {
        printf("Failed to open k4a device!\n");
        return 0;
    }
    else {
        printf("Kinect is switched on!\n");
    }

    // Starting the Device
    if (K4A_RESULT_SUCCEEDED != k4a_device_start_cameras(device, &config))
    {
        printf("Failed to start device\n");
        exit(0);
    }
    else {
        printf("Camera Started\n");
    }

    // Capture a depth frame
    switch (k4a_device_get_capture(device, &capture, TIMEOUT_IN_MS))
    {
    case K4A_WAIT_RESULT_SUCCEEDED:
        printf("Capture Success\n");
        break;
    case K4A_WAIT_RESULT_TIMEOUT:
        printf("Timed out waiting for a capture\n");
        exit(0);
    case K4A_WAIT_RESULT_FAILED:
        printf("Failed to read a capture\n");
        exit(0);
    }

    // Retrieving the capture
    k4a_image_t depth_image = k4a_capture_get_depth_image(capture);

    // Access the depth16 image
    k4a_image_t image = k4a_capture_get_color_image(capture);
    if (image)
    {
        printf(" | Depth16 res:%4dx%4d stride:%5d\n",
            k4a_image_get_height_pixels(image),
            k4a_image_get_width_pixels(image),
            k4a_image_get_stride_bytes(image));

        // Release the image
        k4a_image_release(image);
    }
    else {
        printf("Image is null!\n");
    }

    k4a_image_t ir_image = k4a_capture_get_ir_image(capture);

    if (ir_image)
    {
        printf(" | Depth16 res:%4dx%4d stride:%5d\n",
            k4a_image_get_height_pixels(ir_image),
            k4a_image_get_width_pixels(ir_image),
            k4a_image_get_stride_bytes(ir_image));

        // Release the image
        k4a_image_release(ir_image);
    }
    else {
        printf("Image is null!");
    }

    if (depth_image)
    {
        printf(" | Depth16 res:%4dx%4d stride:%5d\n",
            k4a_image_get_height_pixels(depth_image),
            k4a_image_get_width_pixels(depth_image),
            k4a_image_get_stride_bytes(depth_image));

        // Release the image
        k4a_image_release(depth_image);
    }
    else {
        printf("Image is null!");
    }

    // Release the capture
    k4a_capture_release(capture);

    if (device != NULL)
    {
        k4a_device_close(device);
    }

    return 0;
}

[qm13]

Can you please verify that your USB host controller is one of the supported controllers? See here. Assuming you are using a supported USB host controller then the next step is to provide Azure Kinect logs.

[constantin-ulbrich]

Thank you very much for your help! I have checked the USB host controller and my machine uses one of the supported ones (Intel(R) USB 3.0 eXtensible-Hostcontroller-1.0(Microsoft)). However, unlike on the example image provided in the Docs to check for the USB host controller, my machine's device structure has three nested Generic SuperSpeed USB Hubs. I am not sure if that makes a difference, so I just wanted to mention it to double check. The device connection structure can be viewed in the screenshot below.

Also, I have tested it for every USB port of my PC. The device connection structure always remained the same as shown in the screenshot.

Below, I have generated new Azure Kinect logs, one for each output:

The file named "k4a_output1_capturing_color_image.log" contains the logs, when running the code the first time after plugging in the Kinect. When running the code, I obtain a color image, but no depth- or IR-image.

k4a_output1_capturing_color_image.log

The file named "k4a_output2_capturing_no_color_image.log" contains the logs, when running the code the second time, without having unplugged and reconnnected the Kinect. The output of the code results in a failure to caputre the color-image, but the depth- and IR-image are captured.

k4a_output2_capturing_no_color_image.log

Please, let me know if I should alter the detail level of the logs. Again, thank you very much for your help!

[wes-b]

You are seeing differences in warm / cold startup behavior. You are running with the synchronized_images_only in k4a_device_configuration_t set to false, which means you get all frames, even if we can't sync a color frame to depth/IR frames. If you set this to true, then your captures will always have synchronized color, IR, and depth (depth is not present for passive IR mode). With this setting enabled you can even remove the image NULL checks you have.

[constantin-ulbrich]

Great, thank you very much for your help. This resolved the bug.