Usage of k4a_calibration_2d_to_3d(),

[k123jack]

Want to get (x,y,z) coordinate from color image and having some problems in using k4a_calibration_2d_to_3d() : In order to get depth value of each pixel in the color camera first, I need to call k4a_transformation_depth_image_to_color_camera() right? But this step always get NULL k4a_image_t returned.

Maybe this line is wrong? if (K4A_RESULT_SUCCEEDED!=(k4a_transformation_depth_image_to_color_camera(transformation,depthImage.handle(),outputImage.handle()))){

Whole code is : //-------------------------------------------------------

pragma comment(lib, "k4a.lib")

include <k4a/k4a.h>

include <opencv2/opencv.hpp>

include <k4a/k4a.hpp>

// #include < cstdlib>

include <opencv2/core/core.hpp>

include <opencv2/highgui/highgui.hpp>

include <opencv2/highgui.hpp>

// #include <opencv2/highgui/highgui.hpp> using namespace cv;

include

include

define CV_AA cv::LINE_AA

int main() { k4a::capture capture; const uint32_t device_count = k4a::device::get_installed_count();//查看设备个数 if (device_count == 0) { printf("No K4A devices found\n"); return 0; }

k4a::device device = k4a::device::open(K4A_DEVICE_DEFAULT);

k4a_device_configuration_t config = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL; //设定设备参数
config.color_format = K4A_IMAGE_FORMAT_COLOR_BGRA32;
config.color_resolution = K4A_COLOR_RESOLUTION_720P;
config.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED;
config.camera_fps = K4A_FRAMES_PER_SECOND_30;
config.synchronized_images_only = true;

device.start_cameras(&config);

k4a_calibration_t calibration;
// calibration calib;
if (K4A_RESULT_SUCCEEDED == k4a_device_get_calibration(device.handle(),config.depth_mode,config.color_resolution,&calibration))
{
    printf("calibrate succeed\n");
}
k4a_transformation_t transformation=k4a_transformation_create(&calibration);

k4a::image rgbImage;
k4a::image depthImage;
k4a::image outputImage;

cv::Mat color_frame;
cv::Mat depth_frame;
cv::Mat output_frame;
// Mat color_fram_rect;
int a=0;

Mat img; Mat templ; Mat mask; Mat result;
templ = imread( "/home/ore/Pictures/large.jpg", IMREAD_COLOR );

while (true)
{
    if (device.get_capture(&capture, std::chrono::milliseconds(0)))
    {
        rgbImage = capture.get_color_image();
        depthImage = capture.get_depth_image();

        color_frame = cv::Mat(rgbImage.get_height_pixels(), rgbImage.get_width_pixels(), CV_8UC4, rgbImage.get_buffer());
        depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer());
        depth_frame.convertTo(depth_frame, CV_8U, 1 );
    if (K4A_RESULT_SUCCEEDED!=(k4a_transformation_depth_image_to_color_camera(transformation,depthImage.handle(),outputImage.handle()))){
      printf("convet failed\n" );
    }
    if (outputImage.handle()==NULL){
      printf("conerted img is NULL\n" );
    }
    output_frame = cv::Mat(outputImage.get_height_pixels(),outputImage.get_width_pixels(),CV_16U,outputImage.get_buffer());
    output_frame.convertTo(output_frame,CV_8U,1);

        cv::imshow("color", color_frame);
        cv::imshow("depth", depth_frame);
    cv::imshow("converted",output_frame);

        color_frame.release();
        depth_frame.release();

        capture.reset();
        if (waitKey(1) == 27)
            break;
    }
}
cv::destroyAllWindows();
device.close();
return 0;

}

//------------------- Result : calibrate succeed [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (390): Invalid argument to image_get_width_pixels(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (397): Invalid argument to image_get_height_pixels(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (404): Invalid argument to image_get_stride_bytes(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (383): Invalid argument to image_get_format(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (359): Invalid argument to image_get_buffer(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/transformation/rgbz.c (680): transformation_depth_image_to_color_camera_validate_parameters(). Transformed depth image data is null. [2020-03-23 13:06:21.599] [error] [t=2382] ../src/transformation/transformation.c (710): transformation_depth_image_to_color_camera_validate_parameters( &transformation_context->calibration, &transformation_context->depth_camera_xy_tables, depth_image_data, depth_image_descriptor, custom_image_data, custom_image_descriptor, transformed_depth_image_data, transformed_depth_image_descriptor, transformed_custom_image_data, transformed_custom_image_descriptor) returned failure in transformation_depth_image_to_color_camera_custom() [2020-03-23 13:06:21.599] [error] [t=2382] ../src/sdk/k4a.c (1197): transformation_depth_image_to_color_camera_custom(transformation_handle, depth_image_buffer, &depth_image_descriptor, custom_image_buffer, &dummy_descriptor, transformed_depth_image_buffer, &transformed_depth_image_descriptor, transformed_custom_image_buffer, &dummy_descriptor, interpolation_type, invalid_custom_value) returned failure in k4a_transformation_depth_image_to_color_camera() convet failed conerted img is NULL [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (359): Invalid argument to image_get_buffer(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (390): Invalid argument to image_get_width_pixels(). image_handle ((nil)) is not a valid handle of type k4a_image_t [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (51): k4a_image_t_get_context(). Invalid k4a_image_t (nil) [2020-03-23 13:06:21.599] [error] [t=2382] ../src/image/image.c (397): Invalid argument to image_get_height_pixels(). image_handle ((nil)) is not a valid handle of type k4a_image_t terminate called after throwing an instance of 'cv::Exception' what(): OpenCV(4.2.0) /home/ore/opencv4/opencv-4.2.0/modules/highgui/src/window.cpp:376: error: (-215:Assertion failed) size.width>0 && size.height>0 in function 'imshow'

Aborted (core dumped)

Thx!

[UnaNancyOwen]

It's not a bug in Azure Kinect Sensor SDK. It's a bug in your code.

depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer()); depth_frame.convertTo(depth_frame, CV_8U, 1 ); if (K4A_RESULT_SUCCEEDED != k4a_transformation_depth_image_to_color_camera(transformation,depthImage.handle(),outputImage.handle())){

It's shallow copy. The memory will shared between cv::Mat and k4a::image. Therefore, cv::Mat::convertTo() also rewrites the memory contents of k4a::image. So, you can't passing the correct image data to k4a_transformation_depth_image_to_color_camera() in next line.

- depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer());
+ depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer()).clone();

In your code, you need to make deep copy using cv::Mat::clone() when you create cv::Mat from k4a::image.

---

Please write Markdown correctly. https://guides.github.com/features/mastering-markdown/ Also, please close the problem when it is resolved. (https://github.com/microsoft/Azure-Kinect-Sensor-SDK/issues/1138)

[k123jack]

It's not a bug in Azure Kinect Sensor SDK. It's a bug in your code.

depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer()); depth_frame.convertTo(depth_frame, CV_8U, 1 ); if (K4A_RESULT_SUCCEEDED != k4a_transformation_depth_image_to_color_camera(transformation,depthImage.handle(),outputImage.handle())){

It's shallow copy. The memory will shared between cv::Mat and k4a::image. Therefore, cv::Mat::convertTo() also rewrites the memory contents of k4a::image. So, you can't passing the correct image data to k4a_transformation_depth_image_to_color_camera() in next line.

- depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer());
+ depth_frame = cv::Mat(depthImage.get_height_pixels(), depthImage.get_width_pixels(), CV_16U, depthImage.get_buffer()).clone();

In your code, you need to make deep copy using cv::Mat::clone() when you create cv::Mat from k4a::image.

Please write Markdown correctly. https://guides.github.com/features/mastering-markdown/ Also, please close the problem when it is resolved. (#1138)

Thanks for the reply.

However, depth_frame is working fine with my code, maybe don't need to get a clone. But converting the depth_image to a color_camera always gets NULL k4a_image_t returned.

k4a_transformation_depth_image_to_color_camera(transformation,depthImage.handle(),outputImage.handle())

maybe this line is wrong.

Also tried to create a K4A_IMAGE_FORMAT_DEPTH16 format image as output like here : #893 and got through. Thx.