How to get the RGB pixel value and depth value at the pixel point (x,y) after the depth map and RGB map are aligned?

[Steven-ZY]

How to get the RGB pixel value and depth value at the pixel point (x,y) after the depth map and RGB map are aligned? You know, the sample provided by the SDK just shows us the rendering map, not data after aligning. By the way, I am new to use RGB-D, to be exact, the D435i , I want to know how to combine it with other algorithms such as object detection and pose estimation? Even I can't understand the samples in the SDK? How can I start it? Thanks in advance.

[MartyG-RealSense]

Intel Python tutorial in the link below is a good introduction to aligning depth and RGB data from a bag file, identifying objects from the data and calculating distance from the camera to that object.

https://github.com/IntelRealSense/librealsense/blob/jupyter/notebooks/distance_to_object.ipynb

As mentioned in your other recent case about pose estimation ( https://github.com/IntelRealSense/librealsense/issues/6255 ), if you do not want to use Aruco image tags, OpenCV or ROS then some kind of AI solution will likely be required if you want to calculate the pose of objects.

I did some research into object detection and pose systems in the Python language that might be accessible for someone new to such programs. The link below looks as though it would meet your need for both object detection and pose estimation:

https://github.com/microsoft/singleshotpose

[Steven-ZY]

Dear Sir，you are really a nice man. Thank you so much for your attention to my questions! I just opened the link you suggested, maybe it's suitable for me, I'll have a try. But I’m concentrating on using C++ now. If you have some idea about C++, please let me know, thank you.

[MartyG-RealSense]

You are very welcome. :)

The majority of such programs that I have seen tend to be in the Python language. Of the C++ Deep Neural Network (DNN) object detection programs on the RealSense SDK's examples page, they usually also require use of extra software such as OpenCV or OpenVINO alongside C++.

I did a search for C++ programs with both object detection and pose estimation functions. If you are willing to consider ROS, I found this object detection and pose estimation program for RealSense D435 and robot arms whose labelling indicates that it is C++:

https://github.com/avinashsen707/AUBOi5-D435-ROS-DOPE

[ev-mp]

@Steven-ZY , the rs-align c++ demo shows how to generate Depth to RGB and RGB to Depth alignment. Once established, the (i,j) coordinate provides a direct mapping between Depth and Color frames.

The following phrase is not clear, can you elaborate ?:

You know, the sample provided by the SDK just shows us the rendering map, not data after aligning.

[ev-mp]

@Steven-ZY, do you need more info here? Please update

[MartyG-RealSense]

Case closed due to no further comments received.