Noisy Depth Images of L515

[bhomaidan1990]

• Before opening a new issue, we wanted to provide you with some useful suggestions (Click "Preview" above for a better view):

  • Consider checking out SDK examples.
  • Have you looked in our documentations?
  • Is you question a frequently asked one?
  • Try searching our GitHub Issues (open and closed) for a similar issue.

• All users are welcomed to report bugs, ask questions, suggest or request enhancements and generally feel free to open new issue, even if they haven't followed any of the suggestions above :)

---

Required Info
Camera Model	L515
Firmware Version	01.05.08.01
Operating System & Version	Linux (Ubuntu 18.04)
Kernel Version (Linux Only)	4.9.253-tegra
Platform	NVIDIA Jetson Nano 4G
SDK Version	2.50.0
Language	python
Segment	others

Issue Description

I'm getting a lot of noise while capturing depth images using L515,

My goal (which I'm not able to achieve due to noise) is to identify the depth of the Legos as you can see in the (cropped RGB image):

I'm using Industrial indirect fixed lighting conditions and not sunlight. I'm using this code, and I have tried to use many filters without any significant improvements:

import cv2
import numpy as np
import pyrealsense2 as rs
import time

pipeline = rs.pipeline()
config = rs.config()

# Resolution
res = [(1024, 768), (640, 480), (320, 240)]
resolution = res[0]
print("RealSense Resolution:{}\n".format(resolution))

config.enable_stream(rs.stream.depth, resolution[0], resolution[1], rs.format.z16, 30)
profile = config.resolve(pipeline)
# Start streaming
pipeline.start(config)

# Declare sensor object and set options
depth_sensor = profile.get_device().first_depth_sensor()
depth_sensor.set_option(rs.option.visual_preset, 5) # 5 is short range, 3 is low ambient light
# sensor.set_option(rs.option.laser_power, 30)

depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale = {}".format(depth_scale))

# depth_sensor.set_option(rs.option.confidence_threshold, 2) # 3 is the highest confidence
# depth_sensor.set_option(rs.option.noise_filtering, 1)

try:
    cv2.namedWindow("depth", cv2.WINDOW_NORMAL)
    t0 = time.time()
    # old_depth = np.zeros((resolution[1], resolution[0]))
    # thresh = 100
    while True:
        # Wait for depth frames:
        frames = pipeline.wait_for_frames()
        depth_frame = frames.get_depth_frame()
        if not depth_frame:
            continue

        #------------
        # # FILTERS |
        #------------
        threshold_filter = rs.threshold_filter(min_dist=1.2, max_dist=1.4)
        depth_frame = threshold_filter.process(depth_frame)

        # hole_filling = rs.hole_filling_filter()  # hole filling - hole filling
        # depth_frame = hole_filling.process(depth_frame)

        # spatial_filter = rs.spatial_filter()
        # depth_frame = spatial_filter.process(depth_frame)

        # temporal_filter = rs.temporal_filter(smooth_alpha=0.8, smooth_delta = 80,persistence_control=8)
        # depth_frame = temporal_filter.process(depth_frame)

        # decimation_filter = rs.decimation_filter(4)
        # depth_frame = decimation_filter.process(depth_frame)

        # Convert images to numpy arrays
        depth_array = np.asanyarray(depth_frame.get_data())

        # # Capture
        t1 = time.time()
        if(round(t1-t0)>10):
            print("t0={} , t1={}, int(t1-t0)={}".format(t0, t1, int(t1-t0)))
            cv2.imwrite("depth_{}_.png".format(t0), depth_array)
            t0 = t1
        # # Visualization
        # print("Shape after Filter = ", depth_array.shape)
        cv2.imshow("depth", depth_array)
        if cv2.waitKey(1) & 0xFF == ord('q'):
            break

finally:
    # Stop streaming
    cv2.destroyAllWindows()
    pipeline.stop()

[MartyG-RealSense]

Hi @bhomaidan1990 This case is a follow-up to the RGB image saving that you were performing in https://github.com/IntelRealSense/librealsense/issues/10325 where the camera is looking downwards towards the floor. It also sounds as though you have tried all the L515 presets already ('Short Range', No Ambient Light', 'Low Ambient Light', etc). Is this correct, please?

As Lego has a smooth surface, there may be possible issues with specular reflectivity. On the L515 model, depth quality with reflective surfaces can be negatively impacted if the camera is pointed straight at a surface. Observing the surface with the camera tilted at an angle can help to negate the disruption.

Highly reflective objects such as those with smooth surfaces may have specular reflections that may not get reflected back into the L515's receiver for detection, causing no depth value to be registered. Having the camera positioned above the object and tilted at an angle helps to measure the depth on smooth surfaces.

Placing the camera directly above a highly reflective surface wthout tilting the camera can also cause the depth map to be corrupted.

The source for this information is the official L515 user guide, available from the link below.

https://support.intelrealsense.com/hc/en-us/articles/360051646094-Intel-RealSense-LiDAR-Camera-L515-User-Guide

Could you test whether the depth image improves if you tilt the camera towards the Lego board at an angle, please?

---

Whilst your post-processing filters are not listed in the order recommended by Intel at the link below, as you only have one filter (Threshold) active in your script currently and the rest of the filters are commented out then this should not be causing a problem.

https://dev.intelrealsense.com/docs/post-processing-filters#section-using-filters-in-application-code

---

Also, could you try unplugging the micro-sized end of the USB cable from the base of the camera, reversing the connector's orientation and plugging it back into the camera, please (USB-C cables are two-way insertion at the micro-sized end). Does doing so make any difference?

---

By the way, information at https://github.com/IntelRealSense/librealsense/issues/5393 about a non-RealSense project for identifying Lego pieces using a camera and a neural network may be a helpful reference for your project.

[bhomaidan1990]

Hi @bhomaidan1990 This case is a follow-up to the RGB image saving that you were performing in #10325 where the camera is looking downwards towards the floor. It also sounds as though you have tried all the L515 presets already ('Short Range', No Ambient Light', 'Low Ambient Light', etc). Is this correct, please?

Yes, I have tried all, and the best results came with the short-range option.

As Lego has a smooth surface, there may be possible issues with specular reflectivity. On the L515 model, depth quality with reflective surfaces can be negatively impacted if the camera is pointed straight at a surface. Observing the surface with the camera tilted at an angle can help to negate the disruption.

Is there a recommended value for this angle? 10 20 30 ... degrees

Could you test whether the depth image improves if you tilt the camera towards the Lego board at an angle, please?

I will try ...

Whilst your post-processing filters are not listed in the order recommended by Intel at the link below, as you only have one filter (Threshold) active in your script currently and the rest of the filters are commented out then this should not be causing a problem.

I have tried the filtering one by one.

Also, could you try unplugging the micro-sized end of the USB cable from the base of the camera, reversing the connector's orientation, and plugging it back into the camera, please (USB-C cables are two-way insertion at the micro-sized end). Does doing so make any difference?

I will try this step also., but why does rotating the connector make difference? isn't it supposed to work in both directions?

By the way, information at #5393 about a non-RealSense project for identifying Lego pieces using a camera and a neural network may be a helpful reference for your project.

Thanks for your kind recommendation.

[MartyG-RealSense]

The Short Range preset reduces the values of the Laser Power and Receiver Gain settings. The official L515 user guide states that reducing the Receiver Gain can reduce noise from ambient light, whilst reducing Laser Power can help to avoid the receiver being over-saturated by highly reflective objects.

With the 400 Series stereo depth cameras, a maximum tilt of 30 degrees is usually recommended, beyond which problems may occur in the image. The reflectivity illustration above looks as though it is also suggesting a tilt of around 30 degrees for L515.

The L515 has a known hardware issue with its micro USB port that can negatively affect its operation depending on which direction the USB cable is inserted into the base of the camera. It is not correctable with a software fix. The L515 model is now End of Life (retired), though it continues to receive technical support on the RealSense support forums.

[bhomaidan1990]

Actually, I'm getting good results using the realsense-viewr

when I apply the temporal_filter with alpha=0.1 and delta=75.0 and I reduce the Receiver Gain to the minimum 8, but I couldn't replicate the same results using Python API:

depth_sensor.set_option(rs.option.receiver_gain, 8)
# After Thresholding filter
temporal_filter = rs.temporal_filter(smooth_alpha=0.1, smooth_delta = 75.0,persistence_control=0)
depth_frame = temporal_filter.process(depth_frame)

[MartyG-RealSense]

Checking the post-processing filters in the Viewer, I'm reminded that the L515 camera model only has support for the Temporal filter. Its defaults in the Viewer are '0.40' for Filter Smooth Alpha and '20.000' for Filter Smooth Delta.

It looks as though you also have not got a colorizer configured in your script to colorize the depth data, resulting in a monochrome depth image compared to the Viewer's. https://github.com/IntelRealSense/librealsense/issues/7767#issuecomment-726704671 lists Python references for configuring the colorizer and depth colorization settings.

[bhomaidan1990]

The noise issue can be seen with/without using a colorizer, my issue now is that the Temporal Filter is working perfectly in the realsense-viewer which is not the case with my Python script. Is there any issue in my usage of the temporal_filter? thanks in advance.

[MartyG-RealSense]

I note that you are setting the L515 Temporal filter's persistence_control to '0', which corresponds to disabled (no persistency filtering and no hole-filling). The Viewer has persistence set to 'Valid in 2/last 4' by default, also known as persistency filter setting '3'.

The RealSense SDK's post processing documentation explains that the value '3' corresponds to: "Valid in 2/last 4 - Activated if the pixel was valid in two out of the last 4 frames"

https://dev.intelrealsense.com/docs/post-processing-filters#section-temporal-filter

[bhomaidan1990]

I note that you are setting the L515 Temporal filter's persistence_control to '0', which corresponds to disabled (no persistency filtering and no hole-filling). The Viewer has persistence set to 'Valid in 2/last 4' by default, also known as persistency filter setting '3'.

The RealSense SDK's post processing documentation explains that the value '3' corresponds to: "Valid in 2/last 4 - Activated if the pixel was valid in two out of the last 4 frames"

https://dev.intelrealsense.com/docs/post-processing-filters#section-temporal-filter

I have changed the persistence in my code, and it didn't make any difference, I have the feeling that the temporal_filter is not applied at all using the Python code, because I have played with all the parameters (alpha, delta, persistence), and it didn't change anything!

[MartyG-RealSense]

It may be useful to test a basic example script of setting the Temporal filter in Python to see whether there is an issue somewhere in your script.

import pyrealsense2 as rs
pipe = rs.pipeline()
cfg = rs.config()
cfg.enable_stream(rs.stream.Depth, 640, 480, rs.format.Z16, 30);
pipe.start(cfg)
# Declare filters
temp_filter = rs.temporal_filter()    # Temporal   - reduces temporal noise
frames = pipe.wait_for_frames()
depth_frame = frames.get_depth_frame()
filtered = temp_filter.process(filtered)

On the case that this script came from, there was another Python case linked to it at https://github.com/IntelRealSense/librealsense/issues/10078 where the RealSense user also had non-functioning persistency settings on their Temporal filter even though the Viewer worked normally. In the end they found that a simple one-page script worked, for reasons which they discuss at https://github.com/IntelRealSense/librealsense/issues/10078#issuecomment-1003084771

[bhomaidan1990]

It may be useful to test a basic example script of setting the Temporal filter in Python to see whether there is an issue somewhere in your script.

import pyrealsense2 as rs
pipe = rs.pipeline()
cfg = rs.config()
cfg.enable_stream(rs.stream.Depth, 640, 480, rs.format.Z16, 30);
pipe.start(cfg)
# Declare filters
temp_filter = rs.temporal_filter()    # Temporal   - reduces temporal noise
frames = pipe.wait_for_frames()
depth_frame = frames.get_depth_frame()
filtered = temp_filter.process(filtered)

On the case that this script came from, there was another Python case linked to it at #10078 where the RealSense user also had non-functioning persistency settings on their Temporal filter even though the Viewer worked normally. In the end they found that a simple one-page script worked, for reasons which they discuss at #10078 (comment)

Thanks for your excellent note, the solution was to define the temporal filter outside the While loop as it is temporal and related to the past frames, hence it worked perfectly.

[MartyG-RealSense]

You are very welcome. It's great to hear that you achieved a solution :)