Why do I get a bunch of 0's？ QAQ

[CCCCCCCT]

Required Info
Camera Model	L515
Firmware Version	01.05.08.01
Operating System & Version	{Win (10)
Kernel Version (Linux Only)
Platform	PC
SDK Version	2.50.0
Language	Python
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

Hello, I'm currently developing a function to obtain the depth values of keypoints using the L515 and OpenPose. I've noticed that although the coordinates are correct, my program often gives me depth values of 0, and these frequent 0 values prevent me from proceeding with the next steps.

I’ve carefully reviewed the part of the code where OpenPose captures the keypoints, and it seems correct. However, the L515 frequently returns a depth value of 0.

I would like to ask for advice: Is there something wrong with my code or L515?

Here is the output PersonID 0: Depth value at coordinates (x=291, y=253): 0.0000 m PersonID 0: Depth value at coordinates (x=374, y=231): 0.0000 m PersonID 0: Depth value at coordinates (x=381, y=231): 0.0000 m PersonID 0: Depth value at coordinates (x=384, y=236): 0.0000 m PersonID 0: Depth value at coordinates (x=374, y=236): 0.0000 m PersonID 0: Depth value at coordinates (x=376, y=243): 3.2178 m PersonID 0: Depth value at coordinates (x=376, y=243): 0.0000 m PersonID 0: Depth value at coordinates (x=376, y=241): 3.1893 m PersonID 0: Depth value at coordinates (x=374, y=238): 0.0000 m PersonID 0: Depth value at coordinates (x=371, y=236): 0.0000 m PersonID 0: Depth value at coordinates (x=371, y=233): 0.0000 m PersonID 0: Depth value at coordinates (x=371, y=231): 0.0000 m PersonID 0: Depth value at coordinates (x=369, y=231): 0.0000 m PersonID 0: Depth value at coordinates (x=366, y=231): 0.0000 m

import cv2
import sys
import os
import pyrealsense2 as rs
import numpy as np
import bin.pyopenpose as op

def get_depth_value(depth_frame, x, y, personid):
    if not depth_frame:
        print("No depth frame available")
        return None

    depth_value = depth_frame.get_distance(int(x), int(y))
    print("PersonID {}: Depth value at coordinates (x={}, y={}): {:.4f} m".format(personid ,int(x), int(y), depth_value))
    return depth_value

def main():
    try:
        # Openpose參數
        params = dict()
        params["model_folder"] = "./models/"
        params["net_resolution"] = "256x256"  # 以準確率為代價提升處理速度
        params["model_pose"] = "BODY_25"

        # 初始化OpenPose
        opWrapper = op.WrapperPython()
        opWrapper.configure(params)
        opWrapper.start()

        # 打開相機，連接L515
        cap = cv2.VideoCapture(0,cv2.CAP_DSHOW)
        if not cap.isOpened():
            print("Error: Unable to open camera")
            return

        # 初始化RealSense pipeline
        # 做一次就好
        pipeline = rs.pipeline()
        config = rs.config()
        config.enable_stream(rs.stream.depth)
        # config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

        pipeline.start(config)

        # device = profile.get_device()
        # depth_sensor = device.first_depth_sensor()
        # device.hardware_reset()

        while True:
            ret, frame = cap.read()
            if not ret:
                break

            #L515拿深度
            frames = pipeline.wait_for_frames()
            depth_frame = frames.get_depth_frame()

            #openpose開始處理
            datum = op.Datum()
            datum.cvInputData = frame #cvInputData 是 OpenPose 用於接收輸入影像的屬性。
            datums = op.VectorDatum([datum])
            opWrapper.emplaceAndPop(datums)

            keypoints = datum.poseKeypoints #取得姿態估計的結果，即關鍵點資訊。
            if keypoints is not None:
                for person_id, person in enumerate(keypoints):
                    i = 8  #髖關節
                    x, y, confidence = person[i]
                    if confidence > 0.1:
                        #output_point(i, person_id, x, y, frame)
                        get_depth_value(depth_frame, x, y,person_id)

            pose_frame = datum.cvOutputData
            frame = cv2.addWeighted(frame, 0.7, pose_frame, 0.3, 0)
            #顯示
            cv2.imshow("OpenPose Camera", frame)

            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

        #release
        cap.release()
        pipeline.stop()
        cv2.destroyAllWindows()

    except Exception as e:
        print(f"Error: {e}")
        sys.exit(-1)

if __name__ == "__main__":
    main()

[CCCCCCCT]

Sometimes it can detect the depth value, but most of the time it fails to detect it.

[MartyG-RealSense]

Hi @CCCCCCCT Would it be possible to provide an RGB color image of the scene that the L515 camera is observing in order to see what elements in the scene could be making it difficult for the camera to read depth information from surfaces?

For example, if the surface that is being analyzed is the color black then it will be seen as an area of empty depth (zeroes) unless a strong light source is cast onto that area, as described at https://github.com/IntelRealSense/librealsense/issues/6757

[CCCCCCCT]

The results below the image are generated by my code, and it should have captured my coordinates correctly.

[CCCCCCCT]

Sometimes it detects the coordinates, but sometimes it doesn't."

[MartyG-RealSense]

The RGB color images in the situations where the depth is detected and when it is not detected are very similar, so it does not seem to be a case of the detection being disrupted by elements in the scene.

It also appears that the body's skeleton landmark joints are being detected correctly even on the image where depth is not detected.

I notice that in the image where distance detection is successful you are standing near to the center of the camera view, whilst in the image where distance values are zero you are standing at the far side of the camera view. Being off to the side could make depth detection more difficult for the camera. I recommend trying again whilst in the center of the camera view and see if you have consistent success.

[CCCCCCCT]

Thank you for your help. I'll try again.

[CCCCCCCT]

To prevent it from returning a value of 0, I'm currently summing the depth values of a 5x5 area around the detected point and dividing by the number of non-zero values. However, there are still some cases where it returns 0. This seems to happen more often when the person is on the right side of the frame. Could this be because the resolution of the L515 doesn't match the resolution of my depth image?

leng total: Depth value at coordinates (x=271, y=218): 2.5440 m leng total: Depth value at coordinates (x=349, y=231): 2.1882 m leng total: Depth value at coordinates (x=386, y=231): 2.0876 m leng total: Depth value at coordinates (x=426, y=243): 2.0567 m leng total: Depth value at coordinates (x=461, y=248): 1.9880 m leng total: Depth value at coordinates (x=509, y=256): 1.9010 m leng total: Depth value at coordinates (x=547, y=261): 2.1414 m leng0: Depth value at coordinates (x=549, y=266): 0.0000 m leng total: Depth value at coordinates (x=544, y=266): 2.1384 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng total: Depth value at coordinates (x=544, y=268): 2.1400 m leng total: Depth value at coordinates (x=544, y=266): 2.1435 m leng total: Depth value at coordinates (x=544, y=266): 2.1378 m leng0: Depth value at coordinates (x=537, y=268): 0.0000 m leng0: Depth value at coordinates (x=522, y=268): 0.0000 m leng0: Depth value at coordinates (x=512, y=266): 0.0000 m leng0: Depth value at coordinates (x=524, y=266): 0.0000 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng total: Depth value at coordinates (x=539, y=268): 2.1481 m leng total: Depth value at coordinates (x=542, y=268): 2.1411 m leng0: Depth value at coordinates (x=537, y=268): 0.0000 m leng0: Depth value at coordinates (x=537, y=268): 0.0000 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng0: Depth value at coordinates (x=534, y=268): 0.0000 m leng0: Depth value at coordinates (x=537, y=268): 0.0000 m leng0: Depth value at coordinates (x=539, y=268): 0.0000 m leng0: Depth value at coordinates (x=547, y=268): 0.0000 m leng total: Depth value at coordinates (x=542, y=271): 2.1506 m leng0: Depth value at coordinates (x=502, y=268): 0.0000 m leng total: Depth value at coordinates (x=489, y=261): 3.0071 m leng total: Depth value at coordinates (x=479, y=266): 3.0262 m leng total: Depth value at coordinates (x=476, y=261): 3.0141 m leng0: Depth value at coordinates (x=474, y=263): 0.0000 m leng total: Depth value at coordinates (x=474, y=256): 3.0203 m leng total: Depth value at coordinates (x=476, y=258): 3.0143 m leng total: Depth value at coordinates (x=486, y=246): 3.0043 m leng total: Depth value at coordinates (x=472, y=261): 3.0198 m leng0: Depth value at coordinates (x=451, y=251): 0.0000 m leng total: Depth value at coordinates (x=429, y=246): 2.6605 m leng2: Depth value at coordinates (x=389, y=248): 2.6695 m leng total: Depth value at coordinates (x=356, y=253): 2.2325 m leng total: Depth value at coordinates (x=386, y=251): 2.1657 m leng total: Depth value at coordinates (x=429, y=258): 2.0150 m leng0: Depth value at coordinates (x=436, y=266): 0.0000 m leng total: Depth value at coordinates (x=426, y=288): 2.6229 m

[MartyG-RealSense]

When using the get_distance() instruction, depth values may be correct at the center of the image and become increasingly inaccurate as you move towards coordinates at the sides of the image.

This error would typically occur when using depth-color alignment with get_distance() though. As you are not using alignment and are only using the depth stream, the cause may be something else.

Your depth cfg instruction does not define a resolution and so will use 640x480 by default. Let's try increasing the resolution to the higher supported value of 1024x768 to see whether it makes a difference.

config.enable_stream(rs.stream.depth, 1024, 768, rs.format.z16, 30)

[CCCCCCCT]

Hello, I followed your suggestion and changed that line of code to the one you specified. However, I was already using that line when running my code until I came across some articles and changed it to config_rs.enable_stream(rs.stream.depth). But even after changing it back to what you specified, the depth value still shows up as 0. Perhaps I can try modifying my code to avoid this issue. Or is there anything else I need to adjust in the initialization settings when using config_rs.enable_stream(rs.stream.depth, 1024, 768, rs.format.z16, 30)?

[CCCCCCCT]

When I use config.enable_stream(rs.stream.depth, 1024, 768, rs.format.z16, 30), I notice that the distance in front of the camera always remains around 4.7 meters, whether I move forward or backward, it stays around 4.6 to 4.7 meters. However, when I move to the left side of the screen, the distance starts to decrease, but when I move to the right side, the distance still remains around 4.6 to 4.7 meters. This is clearly incorrect.

[CCCCCCCT]

I also noticed that with config.enable_stream(rs.stream.depth, 1024, 768, rs.format.z16, 30) and config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30), at the same position, I can get depth values in some cases, but not in others.

In the bottom right corner of the screen, I can't get depth values with 640x480, but with 1024x768, I can get them, although they are incorrect. However, they are detected.

[MartyG-RealSense]

Although the L515 camera can depth-sense up to 9 meters, it is best suited to use at 4 meters distance and below.

Are you able to test the camera in the RealSense Viewer tool? If you enable the Stereo Module (the depth stream) and move the mouse cursor over a coordinate on the depth image, the distance of the coordinate in meters is displayed at the bottom of the Viewer window. If depth is measured correctly in the Viewer then that will validate whether or not the depth problem is in your project rather than an issue with the camera.

[CCCCCCCT]

I tried using the RealSense Viewer tool with the resolution set to 640x480, and it was able to detect the depth. Maybe there is an issue with my code. QAQ

[MartyG-RealSense]

It will be difficult to diagnose the code, given the few number of RealSense OpenPose projects to refer to.

It may be worth focusing on the depth value printing line first.

print("PersonID {}: Depth value at coordinates (x={}, y={}): {:.4f} m".format(personid ,int(x), int(y), depth_value)) What function does the {:.4f} part of the above line perform, please? Might it be preventing the depth value from updating correctly?

You could also try disabling the code that deals with zeroes to see whether it could be breaking something.

[CCCCCCCT]

It is a function to output depth values, and it only takes four decimal places.

I discovered a new issue: I found that the coordinates from the images detected by OpenPose do not align with those detected by the L515. So, I want to try aligning them before performing detection.

Additionally, I found that there is an error when I try to specify the size of my color_frame. config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30) "Error: frame didn't arrive within 5000"

[CCCCCCCT]

I found out that the issue was indeed because my depth image and RGB image were not properly aligned. I've successfully resolved it. Thank you! ╰(°▽°)╯

import cv2
import sys
import os
import pyrealsense2 as rs
import numpy as np
import bin.pyopenpose as op

def get_depth_value(depth_image, x, y,person_id):
    #座標在圖像裡面
    if x < 0 or y < 0 or x >= depth_image.shape[1] or y >= depth_image.shape[0]:
        print("Coordinates out of bounds")
        return None

    depth_value = depth_image[y, x]

    if depth_value == 0:
        print(f"Person_id:{person_id},Depth value at coordinates (x={x}, y={y}): 0 m (no depth information)")
    else:
        depth_in_meters = depth_value * 0.00025  #轉成公尺
        print(f"Person_id:{person_id},Depth value at coordinates (x={x}, y={y}): {depth_in_meters:.4f} m")

    return depth_value

def main():
    try:

        # 初始化RealSense pipeline
        pipeline = rs.pipeline()
        config = rs.config()
        config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
        config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

        align_to = rs.stream.color
        align = rs.align(align_to)

        pipeline.start(config)
        #datum = op.Datum()

        # 初始化OpenPose
        params = dict()
        params["model_folder"] = "./models/"
        params["net_resolution"] = "256x256"
        params["model_pose"] = "BODY_25"

        opWrapper = op.WrapperPython()
        opWrapper.configure(params)
        opWrapper.start()

        while True:

            # L515拿深度
            frames = pipeline.wait_for_frames()

            aligned_frames = align.process(frames) 

            aligned_depth_frame = aligned_frames.get_depth_frame()
            aligned_color_frame = aligned_frames.get_color_frame()

            if not aligned_depth_frame or not aligned_color_frame:
                continue

            color_image = np.asanyarray(aligned_color_frame.get_data())
            depth_image = np.asanyarray(aligned_depth_frame.get_data())

            #depth_image = cv2.applyColorMap(cv2.convertScaleAbs(depth_image,alpha=0.03),cv2.COLORMAP_JET)

            datum = op.Datum()
            datum.cvInputData = color_image
            datums = op.VectorDatum([datum])
            opWrapper.emplaceAndPop(datums)

            keypoints = datum.poseKeypoints

            if keypoints is not None:
                #for person in keypoints:
                for person_id, person in enumerate(keypoints):
                    #i = 0
                    nose_x, nose_y = int(person[8][0]), int(person[8][1])
                    #x,y,confidence = person[i][0]
                    #print("x:",nose_x,",y:",nose_y)
                    get_depth_value(depth_image, nose_x, nose_y,person_id)

            #blend_image = cv2.addWeighted(color_image,0.7,depth_image,0.5,0)

            #images = np.hstack((color_image,depth_image))

            pose_frame = datum.cvOutputData
            frame = cv2.addWeighted(color_image, 0.7, pose_frame, 0.3, 0)

            cv2.imshow('window',frame)
            if cv2.waitKey(1) & 0xFF == ord('q'):
                break

            cv2.waitKey(1)

        # Release resources

    finally:
        pipeline.stop()

if __name__ == "__main__":
    main()

[MartyG-RealSense]

It's great to hear that you succeeded. Thanks very much for the update and for sharing your solution!

[CCCCCCCT]

Hello, I would like to test longer distances now. I want to ask, why can't it capture the far end of the wall even though the distance isn't very far?

[MartyG-RealSense]

As you have light entering the room through the windows, please try changing the Preset at the top of the Viewer side-panel from 'Max Range' to 'Low Ambient Light' to see if it makes a difference.

[CCCCCCCT]

I would like to ask if it's possible to use Python to enable this mode?

[MartyG-RealSense]

To set the preset in your Python project, you could try the L515 script provided at https://github.com/IntelRealSense/librealsense/issues/8619 which sets the built-in preset to the Low Ambient Light preset.

depth_sensor.set_option(rs.option.visual_preset, 3) # 5 is short range, 3 is low ambient light

[CCCCCCCT]

Thank you very much for your assistance. I will try again.

[MartyG-RealSense]

Hi @CCCCCCCT Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.