D455 camera calibration and captures feature points

[junmoxiao11]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	D455
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	Linux (Ubuntu 20.04)
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC
SDK Version	{ legacy / 2.<?>.<?> }
Language	python
Segment	Robot

Issue Description

![Uploading a103dbb3f103ecb388efad8b398fff3.png…]() When I was shooting with the D455 camera, I noticed a lot of black depth noise on the image. Is there any way to calibrate the camera for noise reduction? I found the code to capture the feature points, but there's a lot of depth noise on the image. This severely affected the use of the algorithm. So I needed to calibrate the camera. See if the results can be a little more accurate.Is there any way I can learn how to use the D455 camera more?

[MartyG-RealSense]

This kind of image analysis goes beyond what the RealSense SDK on its own is usually capable of. The tool YOLO would typically be combined with a RealSense camera for such tracking of moving objects by analyzing the RGB image, like in the YouTube video at the link below.

https://www.youtube.com/watch?v=--81OoXMvlw

You can find more examples of RealSense + YOLO by googling for the search term position tracking yolo realsense

The next link is an example of someone using RealSense with YOLO to detect people in a bag file.

https://support.intelrealsense.com/hc/en-us/community/posts/1500000482481-Multi-People-Tracking

[junmoxiao11]

Do you know how to install python pyrealsense2 in windows11?

[MartyG-RealSense]

https://github.com/IntelRealSense/librealsense/issues/10315#issuecomment-1068208234 may be a helpful reference.

[junmoxiao11]

Why can I only capture feature points of RGB images with corner tracking algorithm? It cannot capture the feature points in the depth image and read the depth value of these points. In depth images, doesn't the color information reflect the depth information? Can't I just programmatically capture the feature point, read its color information and convert it to depth information?

[MartyG-RealSense]

I researched your question carefully but all of the RealSense corner tracking solutions that I found used the RGB image for depth and not the depth data.

Python examples of RGB-based corner detection that I found are https://github.com/IntelRealSense/librealsense/issues/10484#issuecomment-1142458201 and https://github.com/IntelRealSense/librealsense/issues/12676

If you performed color to depth alignment instead of depth to color then the color image would resize to match to the depth coordinates.

[junmoxiao11]

Do yoou mean that RGB images taken by realsense cameras also contain depth information?

[MartyG-RealSense]

No, what it means is that when aligning depth and color images, the coordinates of the depth image are aligned with the coordinates of the color image to create a single combined depth and color image where both depth and color have the same coordinates. So in an aligned image, you can be confident that if you know a color coordinate then you know the corresponding depth coordinate because they will be the same.

[junmoxiao11]

I read your explanation carefully. Can I understand that by capturing a point in a color image, I get its coordinates. Then the obtained coordinate points are used to read the depth value of the point in the depth image at the same time.

[MartyG-RealSense]

Yes, correct. If the images have been aligned together then a point in the color image will correspond to the same point on the depth image.

[junmoxiao11]

Can I do this in code for a recorded bag file?

[MartyG-RealSense]

Yes, if you use the enable_device_from_file() instruction to use a bag file as the data source instead of a live camera then the script will behave the same as if the data was coming from a camera. The RealSense SDK provides an example Python script called read_bag_example.py at the link below that demonstrates use of this instruction.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/read_bag_example.py#L42

[MartyG-RealSense]

Hi @junmoxiao11 Do you require further assistance with this case, please? Thanks!

[junmoxiao11]

Yes, please do not close this issue. I may have a few more questions for you in the future.

[MartyG-RealSense]

Okay, I will keep the issue open and label it Enhancement as a reminder not to close it.

[junmoxiao11]

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/read_bag_example.py#L42 Hello, in the sample script you provided, the code can only read the deep stream of the bag file. Are there any other examples that can read a RGB stream?

[MartyG-RealSense]

If you want to read both depth and RGB streams from the bag file, you can add the line below beneath line 46 of the script.

config.enable_stream(rs.stream.color, rs.format.bgr8, 30)

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/read_bag_example.py#L46C5-L46C61

[junmoxiao11]

I have modified the code according to your suggestion, but such error still occurs. The script seems to read only the depth stream but not the color stream. But when I open the bag file with the realsense-viewer, there was a color image. Do you have a way to read the color stream in the bag file in python?

[MartyG-RealSense]

A RealSense user who was modifying read_bag_example.py to read the color stream and also had problems found that a solution was to comment out the stream config lines. Their solution is at https://github.com/IntelRealSense/librealsense/issues/9208#issuecomment-860268711

[junmoxiao11]

I found that by changing the bgr8 code to rgb8, I was able to run soomthly. I can now output the coordinates of the captured color feature points. But now I need to output the depth information for these corresponding coordinates as well. Do you know of any relevant examples? Preferably run in python.

[MartyG-RealSense]

It sounds as though the instruction rs2_project_color_pixel_to_depth_pixel would meet your requirements. It converts a single specific RGB color pixel into a depth pixel to return the real-world depth of that pixel. https://github.com/IntelRealSense/librealsense/issues/5603#issuecomment-574019008 has a Python example of this instruction's use.

[junmoxiao11]

I read the depth frame of the video through depth_frame = frames.get_depth_frame(), then defined a function "write_points_to_csv(file_path, frame_number, tracked_objects, depth_frame):" and used "depth = depth_frame.get_distance(point[0], point[1]) writer.writerow([obj.id, point[0], point[1], depth])" to record the depth information as well. But now I have a problem, there are a lot of black noise in the video, resulting in recorded depth information often appears 0. This is the effect I shot. The white paper in the depth image has a lot of black noise. I don't know if this is normal. Do you have any good ways to deal with the noise?

[MartyG-RealSense]

If your ceiling lights are of the fluorescent type then that could create noise in the depth image, as they flicker at a rate that is difficult to see with the human eye.

You could try increasing the FPS to 60 to test whether or not the issue is disruption from the lights' operating frequency.

[MartyG-RealSense]

Hi @junmoxiao11 Do you have an update about this case that you can provide, please? Thanks!

[junmoxiao11]

Is this picture I took normal? Why is there a black noise of about the same area behind the black stick in the depth image? Now I can track the movement of this black stick by tracking the feature points. I output the poisition change information of the tracked points into a file as shown in the figure below: I now want to calibrate the depth camera. That is, the coordinates of the feature points in the image in the file become the world coordinates. So I can figure out its actual motion. Do you know about depth camera calibrate?

[junmoxiao11]

This is my output of the change in position of the feature points. I forgot to add it in the last comment, sorry.

[MartyG-RealSense]

Hi @junmoxiao11 The camera may be having difficulty with reading the black stick simply because it is colored black, for the reason mentioned earlier in this discussion at https://github.com/IntelRealSense/librealsense/issues/12689#issuecomment-1972720201

As I suggested in that comment, you could try enabling the Hole-Filling post-processing filter to fill the empty black hole in.

[junmoxiao11]

Now, I have a more pressing problem: How to calibrate depth cameras? I have been able to capture the featurre point movement information of the black stick in the image and output it, as shown in this figure: Do you know how to turn motion information in pictures into actual displacement? Also the calibration of the camera?

[MartyG-RealSense]

The RealSense SDK has a depth calibration tool called On-Chip built into its firmware. It can be accessed from the 'More' option of the RealSense Viewer. It can also be accessed and controlled with Python code. The SDK has an official Python example of this called depth_auto_calibration_example.py

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/depth_auto_calibration_example.py

Full details of the on-chip calibration tool and other calibration tools can be found here:

https://dev.intelrealsense.com/docs/self-calibration-for-depth-cameras

Regarding displacement, do you mean tracking the motion of the black stick? If you do then https://github.com/IntelRealSense/librealsense/issues/4175 might be a helpful reference.

[MartyG-RealSense]

Hi @junmoxiao11 Do you have an update about this case that you can provide, please? Thanks!

[junmoxiao11]

https://github.com/user-attachments/assets/8b136ca0-6dc0-4898-b858-65169228014a

This is a video I recorded with a wall in the background. Was the video shot well? Why do I feel like the video keeps shaking? That is, the depth at the same point is always changing and there's some noise. Is this all normal?

[MartyG-RealSense]

It does appear as though you have quite a lot of fluctuation in depth values in the same area. Plain walls can be difficult for the camera to read because they lack texture detail.

If you are using the RealSense Viewer then increasing the Laser Power setting to its maximum value of '360' may help. This is because it will make the invisible dot-pattern projection cast onto the wall by the camera more visible to the camera and so increase its ability to extract depth information from the surface.

You could also go to the Post-Processing section of the Viewer's options side-panel and expand open the settings for the Temporal filter. Changing the Temporal filter's 'Filter Smooth Alpha' option from its default value of '0.4' to the lower '0.1' can significantly reduce depth value fluctuation.

[junmoxiao11]

Thanks, after modifying the Settings according to you tips, the video picture has been improved.

[junmoxiao11]

Thanks, after modifying the Settings according to you tips, the video picture has been improved.

[junmoxiao11]

https://github.com/user-attachments/assets/63a4c436-8145-40cb-91e8-021e8bcbf096

As you can see in this video. I can capture the red dot in the video and output its coordinates and depth. Now I want to transform this coordinate information from the image coordinate system to the world coordinate system. That is to calculate the true motion of the red dot.

[MartyG-RealSense]

The instruction rs2_deproject_pixel_to_point would be best in this situation for converting a 2D image pixel to a 3D world point. There is an example of a Python script for doing so at https://github.com/IntelRealSense/librealsense/issues/9749

The script was for the L515 camera model, so you will need to change this line:

config.enable_stream(rs.stream.depth, 1024, 768, rs.format.z16, 30)

to this:

config.enable_stream(rs.stream.depth, 1280, 720, rs.format.z16, 30)