Wrong readings with D435i outdoors

[pepisg]

Hi!

I'm currently using a D435i with an IR filter on an outdoors robot and I'm getting very wrong readings in some scenarios. Here's an example:

As you can see in the camera pointcloud there are some small clusters (labeled NOISE) very close to the axes (the robot's position) which when aligned to the RGB image correspond to a wall that lies much further away, as confirmed by the aligned 2D lidar (labeled as REAL WALL POSITION). I also enclose a video to better grasp the depth of the points

https://user-images.githubusercontent.com/71234974/227645997-34834a45-a0c7-4ab3-9798-1f183365b4fa.mp4

Is there any way of mitigating this behavior either by calibrating the camera or using a post processing filter?

Thanks in advance for your help!

[MartyG-RealSense]

Hi @pepisg As the wall is made up of a row of similar looking vertical sections, this may be causing floating areas of incorrect depth due to a phenomenon called repetitive pattern. This is when the camera's depth sensing can become confused by a horizontal or vertical pattern of similar looking elements in a real-world scene. A RealSense team member offers advice at https://github.com/IntelRealSense/librealsense/issues/6713#issuecomment-651114720 for negating repeitive patterns.

Intel also have a PDF guide about the repetitive pattern phenomenon at the link below.

https://dev.intelrealsense.com/docs/mitigate-repetitive-pattern-effect-stereo-depth-cameras

[MartyG-RealSense]

Hi @pepisg Do you require further assistance with this case, please? Thanks!

[pepisg]

Hi! I have been trying the suggestions in the whitepaper but success has been limited: I have observed some partial improvement but the problem still occurs from time to time.

I tried to enable the HDR and though this seemed to remove the misplaced points, I got an extremely noisy pointcloud overall. Am I missing something on how HDR should be used?

Is there any way to completely solve this problem? or is something to be expected to happen occasionally regardless of the fine tuning you may perform on the camera?

Thanks!

[MartyG-RealSense]

If HDR is enabled then the image will be extremely noisy unless the post-processing filter HDR Merge is enabled.

HDR Merge disabled

HDR Merge enabled

In the RealSense Viewer tool, the HDR Merge filter is enabled by default in the Post Processing list of filters. If you have created your own script and are enabling HDR in it then you will also need to program the HDR Merge filter into your script, as post-processing filters are all disabled by default and have to be deliberately coded into your own scripts. At the link below a RealSense user provides their code for enabling the filter with C++.

https://support.intelrealsense.com/hc/en-us/community/posts/360053181133/comments/360015393773

---

Instead of using HDR, you could try reducing the Second Peak Threshold Advanced Mode setting from its default of '325' to a value of '0'. In the Viewer, this setting can be found in the Stereo Module > Advanced Controls > Depth Control section of the options side-panel.

Second Peak Threshold can either be defined in a custom json camera configuration file or with code, such as a RealSense user's C++ script at https://github.com/IntelRealSense/librealsense/issues/4993

---

If you have the option to change camera models, a new D435 model equipped with IMU and Infrared Pass filter called the D435if was released yesterday. The official information page for this camera model states that it "resolves repetitive pattern and false object detection due to light reflections".

https://github.com/IntelRealSense/librealsense/issues/11640

Shown below are the properties of the NIR filter on the D435if and D435f models, which you can compare to the filter that you are using on your D435i.

[MartyG-RealSense]

Hi @pepisg Do you require further assistance with this case, please? Thanks!

[pepisg]

Hi @MartyG-RealSense. Does the hdr merge work in the same way in the ros wrapper? I tried using it but did not see the results you show in the picture. As for the second peak threshold parameter I already experimented with different values but as I said in my previous comment success was limited: I did see some improvement but the issue did not go away completely. The camera I'm using already has a Clarex NIR-75N filter which got from intel and installed manually on a d435i

[MartyG-RealSense]

My understanding is that the ROS wrapper's implementation of HDR Merge is based on the librealsense one, as the wrapper acts as a ROS compatibility layer with librealsense in the background.

A RealSense team member at https://github.com/IntelRealSense/realsense-ros/issues/2486#issuecomment-1475272809 advises to enable auto_exposure when using hdr_merge in the ROS wrapper.

[SamerKhshiboun]

Hi @pepisg As the wall is made up of a row of similar looking vertical sections, this may be causing floating areas of incorrect depth due to a phenomenon called repetitive pattern. This is when the camera's depth sensing can become confused by a horizontal or vertical pattern of similar looking elements in a real-world scene. A RealSense team member offers advice at IntelRealSense/librealsense#6713 (comment) for negating repeitive patterns.

@pepisg except that recommendation of using enable_auto_exposure while using hdr and hdr_merge, it is recommended to be aligned to latest source code of ros2-developlemnt since it includes hdr and auto exposure fixes.

[pepisg]

Hi @MartyG-RealSense, HDR seems equally noisy for me in the realsense viewer even while having the hdr merge filter enabled:

https://user-images.githubusercontent.com/71234974/231566968-4ea8185a-d1d4-4646-828c-46c8111927ca.mp4

Is there any specific firmware version or librealsense version that I should have?

[MartyG-RealSense]

There is not much that can be done to prevent the depth noise in HDR mode, unfortunately. The settings associated with HDR are mostly intended to stop the infrared stream from flickering.

The infrared flickering can be stopped by setting the Sequence options to 1, 2, 1.

Then go to the Post Processing filter list and enable the HDR Merge filter and set the Filter By Sequnce ID filter to '1'.

Metadata support should also be enabled in the Viewer if it is currently disabled in order to stop infrared from flickering.

---

A couple of ways to close up holes in the depth image include:

1. Enabling the Spatial post-processing filter and setting its Hole Filling setting to 'Unlimited'.

Or:

2. Enabling the Hole Filling post-processing filter.

---

The HDR mode requires firmware 5.12.8.100 or newer and SDK 2.39 or newer.

[pepisg]

Hi @MartyG-RealSense , unfortunately I did not got good results: Spatial filter: Hole filling filter: In both cases HDR merge was enabled. This noise (as well as the one created by repetitive patterns) is quite problematic for my use case because I'm using the realsense depth data for obstacle avoidance so misplaced points cause my robot to avoid obstacles that don't actually exist.

What I'm seing in the realsense viewer is still quite different to your examples. May I be missing something else? Also, would it be useful if I was able to provide you with sample data so you can inspect it?

Is the repetitive pattern problem something that ultimately I will be never able to remove completely?

[MartyG-RealSense]

If you enable the Infrared stream as well as the depth stream, does the infrared image flicker even after setting the sequence ID values like in the images above? If infrared does flicker then it may mean that metadata support needs enabling. If it is currently disabled then when the Viewer is launched, a pop-up will usually appear in the top corner of the Viewer window where the firmware pop-up is, asking if you want to enable metadata.

It looks as though a key difference between your images and mine is that your Viewer is in 3D pointcloud mode, whilst my images were taken in 2D mode. To test the settings in 2D mode, left-click on the '2D' option in the top corner of the Viewer window.

If you can provide a bag file recording of your depth data then I will be happy to check it in my own Viewer.

Also, would it be possible to post an RGB image of your scene into this discussion so that I can look for possible disruptive elements such as repetitive patterns or other environmental factors?

[MartyG-RealSense]

Hi @pepisg Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.