Missing depth pixels

[jrecasens]

Hello,

I’m looking for some advice on an issue I’m facing with my RealSense D400 cameras. I have two D456 cameras mounted outdoors, positioned 5 meters high and angled downward (with no direct sunlight exposure). They’re aimed at a relatively flat surface that’s in motion. However, in the .bag recordings from both cameras, I’m noticing significant gaps in the depth data.

I have been using the HighAccuracy preset (preset 3) which may be the reason for this issue? I assume that .bag recordings have those presets incorporated (As opposed to the post-processing). I am using inn Python:

sensor.set_option(rs.option.visual_preset, 3)

Besides testing the other presets, I’m planning to build a custom preset, but given the wide range of options, I would appreciate some high-level guidance on which parameters are most critical to focus on.

Additionally, could you confirm if the following approach is correct for loading a .json preset?

with open(path_to_settings_file, 'r') as file:
    json_text = file.read().strip()

for device_serial, device in self._enabled_devices.items():
    # Get the active profile and load the JSON file containing settings readable by the RealSense
    device = device.pipeline_profile.get_device()
    advanced_mode = rs.rs400_advanced_mode(device)
    advanced_mode.load_json(json_text)

Thanks!

[MartyG-RealSense]

Hi @jrecasens Changes to the depth data caused by a preset will be included in a bag file recording.

Using the existing Medium Density preset may meet your need for a depth image that does not strip out a lot of detail. It provides a good balance between accuracy and the amount of detail on the image. You can set this preset in your script by changing '3' to '5'.

The values that are configured in the Medium Density preset can be found in the link below:

https://raw.githubusercontent.com/wiki/IntelRealSense/librealsense/d400_presets/MidDensityPreset.json

The most important json parameters in this case are likely to be:

• param-secondpeakdelta which affects how strictly the SDK checks the depth values for confidence in them and excludes values that it does not have high confidence in. The default value is '325'. The Medium Density preset sets this parameter to '600' whilst High Accuracy sets it to '647'.

• param-maxscorethreshb which is 88 in the Medium Density preset and a much larger 2893 in the High Accuracy preset.

In regard to loading a json with Python, your code looks fine. However, code for loading a json from a string typically has errors more often than loading one of the SDK's pre-made presets (which uses a different programming method for doing so). This is why loading an existing SDK preset instead of importing a custom one is preferable if possible.

https://github.com/IntelRealSense/librealsense/issues/6075#issuecomment-600087038 has a good example of a json loading script for Python.

[jrecasens]

Thank you MartyG.

I have been experimenting with different values for those two parameters but the results are still not acceptable. Is there a way to extract (using Python) the depth confidence of each pixel? With that information I could implement my own logic for imputation.

BTW, I was getting a Operation failed with error code=-29 error when loading te custom .json preset, I realized I was not saving the .json End of Line (EOF) as Unix (LF), that solved the issue.

[MartyG-RealSense]

A reference that may be compatible with what you have in mind for developing your own logic in Python is the Software Stereo section of Intel's guide to the basics of stereo depth at the link below.

https://www.intelrealsense.com/stereo-depth-vision-basics/

This section has a Python example script for stereo block-matching. https://github.com/IntelRealSense/librealsense/issues/3185#issuecomment-460540961 describes how stereo cameras rely upon matching of pixels.

import numpy
import cv2

fx = 942.8          # lense focal length
baseline = 54.8     # distance in mm between the two cameras
disparities = 128   # num of disparities to consider
block = 31          # block size to match
units = 0.001       # depth units

sbm = cv2.StereoBM_create(numDisparities=disparities,
blockSize=block)
disparity = sbm.compute(left, right)
depth = np.zeros(shape=left.shape).astype(float)
depth[disparity > 0] = (fx * baseline) / (units * disparity[disparity > 0])

The guide provides a further link to an OpenCV stereo block-matching reference.

https://docs.opencv.org/2.4/modules/calib3d/doc/camera_calibration_and_3d_reconstruction.html?highlight=calib#stereobm

[jrecasens]

Thanks for the material.

However, I'm not looking to code my own block-matching algorithm or reverse engineer the stereoscopic vision algorithm with OpenCV. I want to continue using the depth data provided by Intel RealSense, which I assume utilizes its own proprietary method to obtain depth (or is it as described in the link you provided?).

What I'm particularly interested in is gaining more insight into the underlying depth estimation provided by Intel RealSense cameras. For example, I would like to obtain a NumPy array where, instead of depth values, I receive a confidence score for each pixel. With that information, I can selectively choose depth values based on custom conditions or logic tailored to my application.

[MartyG-RealSense]

The stereo basics guide is the best publicly available reference for how RealSense cameras might analyze pixels for depth accuracy, as the camera's actual stereo algorithms are closed-source and proprietary.

Another couple of json parameters that you could test different values with are Texture Difference Threshold - param-texturedifferencethresh - and Texture Count Threshold - param-texturecountthresh

param-texturedifferencethresh is 0 in the Medium Density preset and 1722 in the High Accuracy preset.

texturecountthresh is 0 in both presets.

https://github.com/IntelRealSense/librealsense/issues/10608#issuecomment-1183341367 provides information about these two parameters, explaining how if texturecountthresh is increased above 0 then it increases the depth filtering effect that param-texturedifferencethresh has on the image.

[jrecasens]

Thanks, Marty. I appreciate your insights and will continue to study these parameters and their impact on my depth maps.

That said, I believe having a built-in depth confidence estimation would be extremely beneficial. If you could share this request with the development team, it would be greatly appreciated.

At the moment, I'm using a custom approach that relies on the coefficient of variation and certain depth threshold limits to identify low-quality regions. However, this method is not ideal. A per-pixel confidence metric, even if it's probabilistic, would significantly enhance the quality of my work.

[MartyG-RealSense]

An Intel guide at the link below provides further information about Texture Difference Threshold, Texture Count Threshold and Second Peak Threshold. It also describes how reducing the value of the medianthreshold parameter from its default of 796 to a lower value such as 625 can provide more valid depth values.

https://dev.intelrealsense.com/docs/depth-map-improvements-for-stereo-based-depth-cameras-on-drones#b-depth-camera-settings

I will pass your suggestion about a per-pixel confidence readout to my Intel RealSense colleagues.

[MartyG-RealSense]

Hi again @jrecasens My Intel RealSense colleagues informed me that there are not plans now to implement per-pixel confidence values.

[jrecasens]

ok, thanks for the update Marty. I realized that "Hole Filling" post-process was making things much worse.

[MartyG-RealSense]

You are very welcome. I'm pleased to hear that you were able to make progress by determining that the hole-filling filter was a problem.

[MartyG-RealSense]

Hi @jrecasens Do you require further assistance with this case, please? Thanks!

[jrecasens]

No, thank you Marty.

[MartyG-RealSense]

Okay, thanks very much for the update. As you do not require further assistance, I will close this case. Thanks again!