About realsense D435i depth map noise

[puyiwen]

---

Required Info
Camera Model	{ D400 }
Firmware Version	(05.14.00.00)
Operating System & Version	Ubuntu20.04
Kernel Version (Linux Only)	5.15.0-58
Platform	PC
SDK Version	2
Language	{/opencv/python/ }
Segment	{ others }

I want to take some depth maps to train depth estimation model and I need indoor data only. However, when I use realsense D435i, I find the depth map has so much noise.

You can see the depth map has a lot of uneven spots, which makes the depth map very unsmooth. I once mentioned my doubts under an issue, but by adjusting the camera parameters, these flashing spots still can't be removed. There are a few things I've tried.

1.Turn IR on or off. 2.Set the frequency to 30HZ/60HZ/90HZ. 3.Debug various resolutions. 4.Turn off auto exposure and adjust the exposure manually myself. 5.Debug all the type of post-processing. 6.Suspected of the influence of fluorescent lamps, turned off fluorescent lamps to continue testing, and found that the spot was still present.

All of the above points have been tried, but the depth map quality has not changed at all. I dont know why, D415 seems better than D435i. Why does the D435i work so poorly? Such a depth map cannot be used as a ground truth.This camera problem has been bothering me for a long time, and I am very desperate for you to help me debug and get a high-quality depth map.

By the way, I would like to be able to get a normal depth map under fluorescent lights. I looked at the manual for debugging realSense under fluorescent lamps, meaning that it needs to be the same frequency as fluorescent lamps. However, I am in Asia, the fluorescent lamp frequency is 50HZ, in the realsense-viewer interface, the frequency is only 30HZ/60HZ/90HZ, and the image resolution of 25HZ does not the size I want. How should I set the frequency so that the depth map quality is less affected by fluorescent lamps?Thank you very much!!

Best wishes

[MartyG-RealSense]

Hi @puyiwen The D415 has around 2x better depth image quality and 2x less noise than the D435 / D435i camera models. The D415 has a smaller field of view size on its depth sensors, which lets less light into the sensors and so the depth noise is lower.

Another difference with D415 is that its projector component is always on, whilst the projector on D435 / D435i pulses in line with exposure. You can enable a setting called Emitter Always On to keep the projector always on so that it replicates the behaviour of the D415's projector.

The camera can be set for regional light operating frequencies of 50 Hz by setting the Power Line Frequency option to 50. If this does not work for you then you can set the FPS speed to the closest matching frequency to the operating frequency of the lights. The nearest FPS to 50 Hz would be 60 FPS, though it would not be an ideal match due to the 10 Hz difference.

You could also try filling in the empty black areas of the image by changing the Depth Unit Scale setting from its default of 0.001 to the smaller value of 0.0001.

[puyiwen]

I tried as you said, the depth map still have uneven dots.

[MartyG-RealSense]

The depth stream (the colored map) does not seem to have dots.

The black and white Infrared Stream below it is meant to have the dots on it.

[puyiwen]

The depth stream (the colored map) does not seem to have dots.

The black and white Infrared Stream below it is meant to have the dots on it.

Well, I saw that the white cabinet and ceiling part behind me had polka dots and it didn't feel smooth. Can I do something to remove this unsmooth part?

[puyiwen]

The depth stream (the colored map) does not seem to have dots.

The black and white Infrared Stream below it is meant to have the dots on it.

I want the depth map like that. The above images from one paper were also taken with Realsense, but I don't know what type of camera they used to take photos. These depth maps are very smooth and can be used as ground truth for depth estimation. Can the D435i achieve this effect?

[puyiwen]

The depth stream (the colored map) does not seem to have dots.

The black and white Infrared Stream below it is meant to have the dots on it.

I set the camera json to Medium Density , the depth seems good, I think I have solve the problem. Thank you for your enthusiastic help again!! Best wishes

[MartyG-RealSense]

You are very welcome! Yes, Medium Density is a very good preset as it provides a good balance between accuracy and depth detail (not having a lot of gaps / holes).

[puyiwen]

You are very welcome! Yes, Medium Density is a very good preset as it provides a good balance between accuracy and depth detail (not having a lot of gaps / holes).

Yes, I agree with you, and I will close the question now.

[puyiwen]

You are very welcome! Yes, Medium Density is a very good preset as it provides a good balance between accuracy and depth detail (not having a lot of gaps / holes).

Sorry to bother you, I get depth and rgb from .bag have some questions.

1. I change the depth unit as 0.01, I want to get depth map with meters, so I divide the depth map by 100. Am I doing right?
2. I use realsense-viewer, and see the depth color map very smooth. However, when I save and visualize the depth, it isnt smooth. Am I doing something wrong?

[MartyG-RealSense]

1. Changing the depth unit scale does not change the Z-distance value of coordinates in meters or their XY values, so it should not be necessary to divide the depth map after changing from the default 0.001 scale to 0.1 scale.

2. Are you playing back the bag file in realsense-viewer or in a script that you have created yourself, please? realsense-viewer applies a range of post-processing filters by default. The filters are not saved into the bag file though. So if the bag is played in a script then no post-processing filters will be applied to it and you have to deliberately program the filters into your script yourself in order to more closely replicate how the bag behaves during playback in realsense-viewer, where it is applying filters to the bag by default.

realsense-viewer is able to apply filters to a bag even though filters are not stored in the bag because it is applying the filters in real-time to the recorded data.

[puyiwen]

1. Changing the depth unit scale does not change the Z-distance value of coordinates in meters or their XY values, so it should not be necessary to divide the depth map after changing from the default 0.001 scale to 0.1 scale.
2. Are you playing back the bag file in realsense-viewer or in a script that you have created yourself, please? realsense-viewer applies a range of post-processing filters by default. The filters are not saved into the bag file though. So if the bag is played in a script then no post-processing filters will be applied to it and you have to deliberately program the filters into your script yourself in order to more closely replicate how the bag behaves during playback in realsense-viewer, where it is applying filters to the bag by default.

realsense-viewer is able to apply filters to a bag even though filters are not stored in the bag because it is applying the filters in real-time to the recorded data.

Well,I get it. I would like to confirm a few things to you.

1. I played back the bag file in realsense-viewer. As you said, the bag file from realsense-viewer applies a range of post-processing filters by default. Do you mean when the bag file saved from realsense-viewer, the camera are set like default.json, not my custom camera set? For example, the Preset changed as Medium Density, the depth unit changed as 0.01. Or do you mean just only post-processing has not be saved?
2. You said the depth map dont need to divide, when I printed one depth map from .bag, and I find the values of the map like [3xx,4xx....]. How can I get the truth depth as meters?

[puyiwen]

1. Changing the depth unit scale does not change the Z-distance value of coordinates in meters or their XY values, so it should not be necessary to divide the depth map after changing from the default 0.001 scale to 0.1 scale.
2. Are you playing back the bag file in realsense-viewer or in a script that you have created yourself, please? realsense-viewer applies a range of post-processing filters by default. The filters are not saved into the bag file though. So if the bag is played in a script then no post-processing filters will be applied to it and you have to deliberately program the filters into your script yourself in order to more closely replicate how the bag behaves during playback in realsense-viewer, where it is applying filters to the bag by default.

realsense-viewer is able to apply filters to a bag even though filters are not stored in the bag because it is applying the filters in real-time to the recorded data.

Well,I get it. I would like to confirm a few things to you.

1. I played back the bag file in realsense-viewer. As you said, the bag file from realsense-viewer applies a range of post-processing filters by default. Do you mean when the bag file saved from realsense-viewer, the camera are set like default.json, not my custom camera set? For example, the Preset changed as Medium Density, the depth unit changed as 0.01. Or do you mean just only post-processing has not be saved?
2. You said the depth map dont need to divide, when I printed one depth map from .bag, and I find the values of the map like [3xx,4xx....]. How can I get the truth depth as meters?

[puyiwen]

1. Changing the depth unit scale does not change the Z-distance value of coordinates in meters or their XY values, so it should not be necessary to divide the depth map after changing from the default 0.001 scale to 0.1 scale.
2. Are you playing back the bag file in realsense-viewer or in a script that you have created yourself, please? realsense-viewer applies a range of post-processing filters by default. The filters are not saved into the bag file though. So if the bag is played in a script then no post-processing filters will be applied to it and you have to deliberately program the filters into your script yourself in order to more closely replicate how the bag behaves during playback in realsense-viewer, where it is applying filters to the bag by default.

realsense-viewer is able to apply filters to a bag even though filters are not stored in the bag because it is applying the filters in real-time to the recorded data.

Well,I get it. I would like to confirm a few things to you.

1. I played back the bag file in realsense-viewer. As you said, the bag file from realsense-viewer applies a range of post-processing filters by default. Do you mean when the bag file saved from realsense-viewer, the camera are set like default.json, not my custom camera set? For example, the Preset changed as Medium Density, the depth unit changed as 0.01. Or do you mean just only post-processing has not be saved?
2. You said the depth map dont need to divide, when I printed one depth map from .bag, and I find the values of the map like [3xx,4xx....]. How can I get the truth depth as meters?

[MartyG-RealSense]

1. When a bag file is recorded in realsense-viewer, the stream data and some (but not all) of the current settings are saved to the bag file. Examples of values that are saved and automatically applied to the bag playback are exposure, gain and depth scale.

I tested recording a bag with a custom-created preset (not one of the built-in ones) and found that when the bag was loaded, the preset was set to '0' (Custom) instead of the custom preset.

However, if a custom preset had been applied before recording started then I would expect that some of the settings that were adjusted by the preset would have their updated values saved into the bag.

2. When the depth scale is changed, it is usually not necesssary to make any manual adjustment to the depth map to compensate for the scale change.

0.01 is a scale that is not normally used with 400 Series cameras, except for the D405 close-range model. The default scale of 0.001 is usually only changed downwards to 0.0001 instead of upwards to 0.01. So there are no previous references to explain the depth results that you are getting, unfortunately. As mentioned above though, if 0.01 is used for a D435i in realsense-viewer and the depth coordinate is checked then the distance value in meters that is displayed in the bottom corner of the depth stream panel is the same at 0.01 as it is at 0.001.

[puyiwen]

1. When a bag file is recorded in realsense-viewer, the stream data and some (but not all) of the current settings are saved to the bag file. Examples of values that are saved and automatically applied to the bag playback are exposure, gain and depth scale.

I tested recording a bag with a custom-created preset (not one of the built-in ones) and found that when the bag was loaded, the preset was set to '0' (Custom) instead of the custom preset.

However, if a custom preset had been applied before recording started then I would expect that some of the settings that were adjusted by the preset would have their updated values saved into the bag.

2. When the depth scale is changed, it is usually not necesssary to make any manual adjustment to the depth map to compensate for the scale change.

0.01 is a scale that is not normally used with 400 Series cameras, except for the D405 close-range model. The default scale of 0.001 is usually only changed downwards to 0.0001 instead of upwards to 0.01. So there are no previous references to explain the depth results that you are getting, unfortunately. As mentioned above though, if 0.01 is used for a D435i in realsense-viewer and the depth coordinate is checked then the distance value in meters that is displayed in the bottom corner of the depth stream panel is the same at 0.01 as it is at 0.001.

Well, the camera set has be saved into the bag file. About depth unit set as 0.01, because I find when I set 0.01, the depth color map seems very smooth which can be able as depth estimation ground truth. I also change the value as 0.001 and 0.0001, and I find the depth color map still have uneven spots, so I set the depth unit as 0.01. I create my post-processing script with .bag file, the depth map seems smooth like realsense-viewer. However, when I declare decimation = rs.decimation_filter(), the depth map shape is changed as (160, 216), not the original depth map shape (640,480). How can I change the post-processing depth map shape as original depth map shape?

[MartyG-RealSense]

The decimation post-processing filter supports scaling values in the range 2-8, with the value that is set determining what the resolution will be divided by to produce a lower resolution. The default value is '2'.

So if a decimation filter is defined (with a default 'magnitude' value of 2) and the stream resolution is 1280x720 then the downscaled resolution should be 640x360.

[puyiwen]

The decimation post-processing filter supports scaling values in the range 2-8, with the value that is set determining what the resolution will be divided by to produce a lower resolution. The default value is '2'.

So if a decimation filter is defined (with a default 'magnitude' value of 2) and the stream resolution is 1280x720 then the downscaled resolution should be 640x360.

Thank you for your reply! The post-processing I think I have got it. About get depth map with meters. I use the code profile =pipeline.start(config) depth_sensor = profile.get_device().first_depth_sensor() depth_scale = depth_sensor.get_depth_scale() to get the depth_scale, and then I use the code depth_image*depth_scale to get the depth map with meters. The depth_image is the depth map from bag file. Am I doing right?

[MartyG-RealSense]

Yes, the real-world distance in meters is obtained by multiplying the 16-bit pixel depth value (uint16_t) by the depth scale. For example, if the 16-bit pixel depth value was 6500 and it was multiplied by the D435i's default depth scale value of 0.001 then the real-world distance in meters would be 6.5 meters.

[MartyG-RealSense]

Surfaces that have strong reflections on them like the wall in the first RGB image and the wall rail in the 4th RGB image will be more difficult for the camera to analyze for depth information. These reflections can be dampened to improve the image if a physical filter product called a linear polarization filter is purchased and attached over the lenses on the outside of the camera. These are available affordably from stores such as Amazon by searching for the term linear polarizing film sheet

Surfaces that are plain / smooth and have low texture detail on them will also be more difficult to read at far distance from the camera, where the camera's built-in infrared dot pattern projection will have difficulty reaching them. Examples of low-texture surfaces are plain walls, doors, chairs and desks, and the cardboard boxes. Increasing the camera's Laser Power setting to its maximum value of '360' or using an external pattern projector with a higher power output / range can help with this.

Fluorescent lights such as the ceiling strip-lights in RGB image 2 can also cause interference, as they contain hot gases that flicker at frequencies that are difficult to see with the human eye. They can be compensated for by setting the RGB option Power Line Frequency to 30, 50 or 60 (Hz) depending on what the power frequency is in your country. For example, in Europe it is typically 50, whilst North America is 60. The link below has a list of the frequencies for each country.

https://en.wikipedia.org/wiki/Mains_electricity_by_country

If your country region is 60 then setting the FPS speed to 60 can also help to reduce interference from fluorescent ceiling strip lights.

[puyiwen]

Surfaces that have strong reflections on them like the wall in the first RGB image and the wall rail in the 4th RGB image will be more difficult for the camera to analyze for depth information. These reflections can be dampened to improve the image if a physical filter product called a linear polarization filter is purchased and attached over the lenses on the outside of the camera. These are available affordably from stores such as Amazon by searching for the term linear polarizing film sheet

Surfaces that are plain / smooth and have low texture detail on them will also be more difficult to read at far distance from the camera, where the camera's built-in infrared dot pattern projection will have difficulty reaching them. Examples of low-texture surfaces are plain walls, doors, chairs and desks, and the cardboard boxes. Increasing the camera's Laser Power setting to its maximum value of '360' or using an external pattern projector with a higher power output / range can help with this.

Fluorescent lights such as the ceiling strip-lights in RGB image 2 can also cause interference, as they contain hot gases that flicker at frequencies that are difficult to see with the human eye. They can be compensated for by setting the RGB option Power Line Frequency to 30, 50 or 60 (Hz) depending on what the power frequency is in your country. For example, in Europe it is typically 50, whilst North America is 60. The link below has a list of the frequencies for each country.

https://en.wikipedia.org/wiki/Mains_electricity_by_country

If your country region is 60 then setting the FPS speed to 60 can also help to reduce interference from fluorescent ceiling strip lights.

Thank you very much! I increase the camera's Laser Power to the value '360', but the depth color map still have noise. I want to use the post-processing to improve the quality of the depth. I have used decimation_filter->spatial_filter. Can you give me some other post-processing advice?

[MartyG-RealSense]

If depth is fluctuating then you may be able to stabilize it by defining a temporal filter with a Smooth Alpha value of 0.1. An example of defining this filter in Python is at https://github.com/IntelRealSense/librealsense/issues/10963#issuecomment-1268093845

You could also calibrate your camera to check whether the problem areas of the image are caused by mis-calibration. This can be done with the On-Chip Calibration tool that can be found under the 'More' option at the top of the RealSense Viewer's options side-panel.

[puyiwen]

If depth is fluctuating then you may be able to stabilize it by defining a temporal filter with a Smooth Alpha value of 0.1. An example of defining this filter in Python is at #10963 (comment)

You could also calibrate your camera to check whether the problem areas of the image are caused by mis-calibration. This can be done with the On-Chip Calibration tool that can be found under the 'More' option at the top of the RealSense Viewer's options side-panel.

Hi, I have solved the question of depth map quality. And I also need imu data. What can I do when I use Realsense-viewer to record the bag file?

[MartyG-RealSense]

If the Motion Module is already enabled when you click the Record button then the IMU data and any other streams that are enabled will be recorded into the bag.

Streams that are enabled after recording begins will not be included in the bag recording.

[MartyG-RealSense]

Do your edges become sharper if you disable the Decimation post-processing filter in the RealSense Viewer by left-clicking on the blue icon beside it to turn it to red (off) please?

[MartyG-RealSense]

This may be a distance-related limitation. As the distance of an observed surface / object from the camera increases, depth measurement error, and therefore depth noise, increases linearly over distance. This phenomenon is known as RMS Error. On a D435i, this error will typically begin to become noticable at around 3 meters from the camera and continue to increase as the distance of the observed area from the camera becomes greater.

The D455 camera model has 2x the accuracy over distance of the D435i model, meaning that at 6 meters the D455 has the same depth measuring accuracy that the D435i has at 3 meters.