search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.49k stars 620 forks source link

Unable to get color images using custom code #1653

Open PucklaJ opened 3 years ago

PucklaJ commented 3 years ago

Describe the bug I want to write some code which uses the SDK, but for some reason I am unable to get a color image from the camera, but the Kinect Azure Viewer works completely fine.

To Reproduce Here is a code sample, which uses SDL to display the image:

#include <SDL2/SDL.h>
#include <SDL2/SDL2_framerate.h>
#include <cassert>
#include <iostream>
#include <k4a/k4a.hpp>
#include <k4abt.hpp>

int main(int args, char *argv[]) {
  assert(SDL_Init(SDL_INIT_VIDEO) == 0);

  auto *window =
      SDL_CreateWindow("NoColorFrame", SDL_WINDOWPOS_CENTERED,
                       SDL_WINDOWPOS_CENTERED, 1280, 720, SDL_WINDOW_SHOWN);
  assert(window);
  auto *renderer = SDL_CreateRenderer(window, -1, SDL_RENDERER_ACCELERATED);
  assert(renderer);
  auto *texture = SDL_CreateTexture(renderer, SDL_PIXELFORMAT_YUY2,
                                    SDL_TEXTUREACCESS_STREAMING, 1280, 720);
  assert(texture);
  FPSmanager fps_manager;
  SDL_initFramerate(&fps_manager);
  SDL_setFramerate(&fps_manager, 30);

  auto device(k4a::device::open(K4A_DEVICE_DEFAULT));
  k4a_device_configuration_t device_config = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
  device_config.camera_fps = K4A_FRAMES_PER_SECOND_30;
  device_config.color_format = K4A_IMAGE_FORMAT_COLOR_YUY2;
  device_config.color_resolution = K4A_COLOR_RESOLUTION_720P;
  device_config.synchronized_images_only = false;
  device_config.depth_mode = K4A_DEPTH_MODE_OFF;
  device.start_cameras(&device_config);

  bool running{true};
  SDL_Event event;

  while (running) {
    while (SDL_PollEvent(&event)) {
      switch (event.type) {
      case SDL_QUIT:
        running = false;
        break;
      }
    }

    {
      k4a::capture capture;
      if (device.get_capture(&capture, std::chrono::milliseconds(1))) {
        if (const auto color(capture.get_color_image()); color) {
          assert(SDL_UpdateTexture(texture, nullptr, color.get_buffer(),
                                   color.get_stride_bytes()) == 0);
        }
      }
    }

    SDL_RenderCopy(renderer, texture, nullptr, nullptr);
    SDL_RenderPresent(renderer);

    SDL_framerateDelay(&fps_manager);
  }

  SDL_DestroyTexture(texture);
  SDL_DestroyRenderer(renderer);
  SDL_DestroyWindow(window);
  SDL_Quit();
  return 0;
}

Expected behavior I expect a window showing a live feed of the camera.

Logs These are the logs with K4A_LOG_LEVEL set to t:

[2021-08-02 11:42:12.836] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (53): k4a_device_t_create(). Created   k4a_device_t 00000176854486C0
[2021-08-02 11:42:12.837] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth_mcu\depth_mcu.c (29): depthmcu_t_create(). Created   depthmcu_t 00000176853FAC80
[2021-08-02 11:42:12.837] [trace] [t=9428] d:\a\1\s\extern\azure-kinect-sensor-sdk\src\usbcommand\usb_cmd_priv.h (101): usbcmd_t_create(). Created   usbcmd_t 000001768544A0F0
[2021-08-02 11:42:12.847] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (343): find_libusb_device(). Container ID found: {d2ad55db-d75d-4f40-a990-a5790837c5a9} 
[2021-08-02 11:42:12.847] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (224): populate_serialnumber(). Serial Number found 000420192912
[2021-08-02 11:42:12.847] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=00000115, PayloadSize=255
[2021-08-02 11:42:12.862] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\color_mcu\color_mcu.c (41): colormcu_t_create(). Created   colormcu_t 00000176853B60F0
[2021-08-02 11:42:12.862] [trace] [t=9428] d:\a\1\s\extern\azure-kinect-sensor-sdk\src\usbcommand\usb_cmd_priv.h (101): usbcmd_t_create(). Created   usbcmd_t 0000017685449D70
[2021-08-02 11:42:12.868] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (343): find_libusb_device(). Container ID found: {d2ad55db-d75d-4f40-a990-a5790837c5a9} 
[2021-08-02 11:42:12.869] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (224): populate_serialnumber(). Serial Number found 000420192912
[2021-08-02 11:42:12.869] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\calibration\calibration.c (47): calibration_t_create(). Created   calibration_t 00000176894D1010
[2021-08-02 11:42:12.869] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=00000111, PayloadSize=10240
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (54): capturesync_t_create(). Created   capturesync_t 0000017682836950
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_create(). Created   queue_t 00000176894E4200       
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_create(). Created   queue_t 00000176894E5100       
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_create(). Created   queue_t 00000176894E4480       
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (51): depth_t_create(). Created   depth_t 00000176853C4600       
[2021-08-02 11:42:12.891] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=00000201, PayloadSize=18 
[2021-08-02 11:42:12.894] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=00000201, PayloadSize=18 
[2021-08-02 11:42:12.900] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (225): ******************** Device Info ********************      
[2021-08-02 11:42:12.900] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (226): K4A SDK version:     1.4.1
[2021-08-02 11:42:12.901] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=00000115, PayloadSize=255
[2021-08-02 11:42:12.925] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (232): Serial Number:       000420192912
[2021-08-02 11:42:12.925] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (236): RGB Sensor Version:  1.6.98
[2021-08-02 11:42:12.925] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (239): Depth Sensor Version:1.6.70
[2021-08-02 11:42:12.925] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (242): Mic Array Version:   1.6.14
[2021-08-02 11:42:12.926] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (245): Sensor Config:       6109.7
[2021-08-02 11:42:12.926] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (248): Build type:          Release
[2021-08-02 11:42:12.926] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (253): Signature type:      MSFT
[2021-08-02 11:42:12.926] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (259): ****************************************************
[2021-08-02 11:42:12.926] [warning] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (88): is_fw_version_compatable(). Firmware version for RGB is 1.6.98. Consider upgrading to 1.6.110 or newer. 
[2021-08-02 11:42:12.926] [warning] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (88): is_fw_version_compatable(). Firmware version for Depth is 1.6.70. Consider upgrading to 1.6.79 or newer.
[2021-08-02 11:42:12.926] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dewrapper\dewrapper.c (57): dewrapper_t_create(). Created   dewrapper_t 00000176853F5450
[2021-08-02 11:42:12.926] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_create(). Created   queue_t 00000176894E5010
[2021-08-02 11:42:12.926] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=000000f2, PayloadSize=0
[2021-08-02 11:42:12.931] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=0000000a, PayloadSize=0
[2021-08-02 11:42:12.935] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\color\color.cpp (45): color_t_create(). Created   color_t 0000017685476C60
[2021-08-02 11:42:12.961] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\imu\imu.c (70): imu_t_create(). Created   imu_t 00000176894F04E0
[2021-08-02 11:42:12.961] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_create(). Created   queue_t 00000176894E6690
[2021-08-02 11:42:12.961] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (653): usb_cmd_io(). XFR: Cmd=80000004, PayloadSize=0
[2021-08-02 11:42:12.961] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (850): k4a_device_start_cameras(). k4a_device_start_cameras starting
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (868): k4a_device_start_cameras(). Starting camera's with the following config.
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (869): k4a_device_start_cameras().     color_format:2
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (870): k4a_device_start_cameras().     color_resolution:1
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (871): k4a_device_start_cameras().     depth_mode:0
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (872): k4a_device_start_cameras().     camera_fps:2
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (873): k4a_device_start_cameras().     synchronized_images_only:0
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (874): k4a_device_start_cameras().     depth_delay_off_color_usec:0
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (875): k4a_device_start_cameras().     wired_sync_mode:0
[2021-08-02 11:42:12.961] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (876): k4a_device_start_cameras().     subordinate_delay_off_master_usec:0
[2021-08-02 11:42:12.962] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (877): k4a_device_start_cameras().     disable_streaming_indicator:0
[2021-08-02 11:42:12.962] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\usbcommand\usbcommand.c (649): usb_cmd_io(). XFR: Cmd=80000001, CmdLength=13, PayloadSize=0, CmdData=00000000 00000000...
[2021-08-02 11:42:12.973] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (916): k4a_device_start_cameras(). k4a_device_start_cameras started
[2021-08-02 11:42:15.133] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_destroy(). Destroyed queue_t 00000176894E6690
[2021-08-02 11:42:15.133] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\imu\imu.c (70): imu_t_destroy(). Destroyed imu_t 00000176894F04E0
[2021-08-02 11:42:15.795] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\color\color.cpp (45): color_t_destroy(). Destroyed color_t 0000017685476C60
[2021-08-02 11:42:15.796] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_destroy(). Destroyed queue_t 00000176894E5010
[2021-08-02 11:42:15.797] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dewrapper\dewrapper.c (57): dewrapper_t_destroy(). Destroyed dewrapper_t 00000176853F5450
[2021-08-02 11:42:15.797] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth\depth.c (51): depth_t_destroy(). Destroyed depth_t 00000176853C4600
[2021-08-02 11:42:15.798] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_destroy(). Destroyed queue_t 00000176894E4200
[2021-08-02 11:42:15.799] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_destroy(). Destroyed queue_t 00000176894E5100
[2021-08-02 11:42:15.799] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\queue\queue.c (39): queue_t_destroy(). Destroyed queue_t 00000176894E4480
[2021-08-02 11:42:15.800] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (54): capturesync_t_destroy(). Destroyed capturesync_t 0000017682836950
[2021-08-02 11:42:15.801] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\calibration\calibration.c (47): calibration_t_destroy(). Destroyed calibration_t 00000176894D1010
[2021-08-02 11:42:15.802] [trace] [t=9428] d:\a\1\s\extern\azure-kinect-sensor-sdk\src\usbcommand\usb_cmd_priv.h (101): usbcmd_t_destroy(). Destroyed usbcmd_t 000001768544A0F0
[2021-08-02 11:42:15.802] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\depth_mcu\depth_mcu.c (29): depthmcu_t_destroy(). Destroyed depthmcu_t 00000176853FAC80
[2021-08-02 11:42:15.805] [trace] [t=9428] d:\a\1\s\extern\azure-kinect-sensor-sdk\src\usbcommand\usb_cmd_priv.h (101): usbcmd_t_destroy(). Destroyed usbcmd_t 0000017685449D70
[2021-08-02 11:42:15.805] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\color_mcu\color_mcu.c (41): colormcu_t_destroy(). Destroyed colormcu_t 00000176853B60F0
[2021-08-02 11:42:15.805] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\sdk\k4a.c (53): k4a_device_t_destroy(). Destroyed k4a_device_t 00000176854486C0
[2021-08-02 11:42:15.807] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dynlib\dynlib_windows.c (22): dynlib_t_create(). Created   dynlib_t 000001768549AF50
[2021-08-02 11:42:15.810] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dynlib\dynlib_windows.c (131): dynlib_create(). Depth Engine loaded C:\Projekte\k4abt_no_color_image\build\windows\x64\debug\depthengine_2_0.DLL
[2021-08-02 11:42:15.810] [info] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\deloader\deloader.cpp (51): verify_plugin(). Loaded Depth Engine version: 2.1.3
[2021-08-02 11:42:15.811] [trace] [t=9428] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\dynlib\dynlib_windows.c (22): dynlib_t_destroy(). Destroyed dynlib_t 000001768549AF50

Desktop (please complete the following information):

Additional context If I enable depth I get a lot of these lines:

[2021-08-02 11:44:47.598] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1527733 type:Depth
[2021-08-02 11:44:47.632] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1561055 type:Depth
[2021-08-02 11:44:47.665] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1594400 type:Depth
[2021-08-02 11:44:47.699] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1627722 type:Depth
[2021-08-02 11:44:47.732] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1661066 type:Depth
[2021-08-02 11:44:47.765] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1694400 type:Depth
[2021-08-02 11:44:47.799] [error] [t=16316] D:\a\1\s\extern\Azure-Kinect-Sensor-SDK\src\capturesync\capturesync.c (142): replace_sample(). capturesync_drop, releasing capture early due to full queue TS:   1727733 type:Depth
PucklaJ commented 3 years ago

I just tried to build the SDK from source to try out the examples. It seems that all examples including the tests have the same issue. There is no color image and I get the same releasing capture early logs.

PucklaJ commented 3 years ago

Every Kinect Azure application I compile is unable to get a color image. But every precompiled application I got when downloading the SDK works as expected

qm13 commented 3 years ago

Can you confirm that the k4aviewer and k4abt_simple_3d_viewer included /tools in the installer work?

PucklaJ commented 3 years ago

Yes they both work completely fine. Only the programs which I compile myself do have this problem