`depth_trunc` parameter ignored when calling `o3d.geometry.PointCloud.create_from_depth_image`

[fjulian]

Checklist

• [X] I have searched for similar issues.
• [X] For Python issues, I have tested with the latest development wheel.
• [X] I have checked the release documentation and the latest documentation (for master branch).

Describe the issue

When trying to compute a cloud based on a depth image, the depth_trunc parameter of the function o3d.geometry.PointCloud.create_from_depth_image seems to be ignored. In the point cloud, there are points at where the far plane would be, i.e. further than the specified truncation. In the code snippet below, depth_trunc is set to far, but also changing this to lower values, e.g. 1.0, doesn't change the resulting point cloud. Is this a mistake/misunderstanding in how I use the function or a bug?

A work around is to remove the depth_trunc value altogether, and instead include the line depth = np.where(depth < far - 0.01, depth, 0.0) which is currently commented out in the snippet below. However, it would be nice to use depth_trunc instead of this "hack".

Steps to reproduce the bug

import pybullet as pb
import pybullet_data
import matplotlib.pyplot as plt
import open3d as o3d
import numpy as np

if __name__ == "__main__":

    pb.connect(pb.DIRECT)
    pb.setAdditionalSearchPath(pybullet_data.getDataPath())
    pb.loadURDF("plane.urdf")
    pb.loadURDF("table/table.urdf", [0, 0, 0])

    near = 0.01
    far = 4.0
    width = 120
    height = 100
    fov_x = 60
    aspect = width / height
    pm = pb.computeProjectionMatrixFOV(
        fov=fov_x, aspect=aspect, nearVal=near, farVal=far
    )
    vm = pb.computeViewMatrix((0, -2, 1.5), (0, 1, 0), (0, 0, 1))
    w, h, rgb, d, mask = pb.getCameraImage(width, height, vm, pm)

    rgb = rgb[:, :, :3]

    # Visualize rbg image
    plt.imshow(rgb)
    plt.show()

    # Get depth image
    depth = 1.0 * near * far / (far - (far - near) * d)
    # depth = np.where(depth < far - 0.01, depth, 0.0)

    plt.imshow(depth)
    plt.show()

    # Convert to open3d
    o3d_depth = o3d.geometry.Image(depth)
    fx = width / (2 * np.tan(np.deg2rad(fov_x) / 2))
    fy = fx / aspect
    o3d_intrinsics = o3d.camera.PinholeCameraIntrinsic(
        width, height, fx, fy, width / 2, height / 2
    )

    point_cloud = o3d.geometry.PointCloud.create_from_depth_image(
        depth=o3d_depth, intrinsic=o3d_intrinsics, depth_scale=1.0, depth_trunc=far
    )
    o3d.visualization.draw_geometries([point_cloud])

    pb.disconnect()

Error message

No response

Expected behavior

No response

Open3D, Python and System information

- Operating system: Ubuntu 20.04
- Python version: Python 3.8.10
- Open3D version: 0.17.0, also tried with 0.15.2
- System architecture: x86
- Is this a remote workstation?: no
- How did you install Open3D?: pip
- Compiler version (if built from source): n/a

Additional information

No response

[saurabheights]

Please carefully check your data, its current values (maybe units are wrong) or your code.

See sample code below to show proper clipping.

import numpy as np
import open3d as o3d

if __name__ == "__main__":
    print("Read Redwood dataset")

    dataset = o3d.data.SampleRedwoodRGBDImages()

    rgbd_images = []
    depth_raw = o3d.io.read_image(dataset.depth_paths[0])
    color_raw = o3d.io.read_image(dataset.color_paths[0])

    d = np.asarray(depth_raw)
    non_zero_d = d[np.nonzero(d)]
    print("Number of depth values less than 1 meter are ", np.count_nonzero(non_zero_d < 1000))

    pcd = o3d.geometry.PointCloud.create_from_depth_image(
        depth_raw,
        o3d.camera.PinholeCameraIntrinsic(
            o3d.camera.PinholeCameraIntrinsicParameters.PrimeSenseDefault
        ),
        np.identity(4),
        depth_scale=1000.0,
        depth_trunc=1,
    )
    print(pcd)
    pcd.transform([[1, 0, 0, 0], [0, -1, 0, 0], [0, 0, -1, 0], [0, 0, 0, 1]])
    o3d.visualization.draw_geometries([pcd])

which gives output

Read Redwood dataset
Number of depth values less than 1 meter are  4828
PointCloud with 4828 points.

Image using depth_trunc of 1.5 meter to show chair getting clipped:-

[theNded]

Would you like to share this depth in a npy file: depth = 1.0 * near * far / (far - (far - near) * d)? That will help me identify the problem. Thank you!

[fjulian]

Thanks for looking into this!

@theNded I attached the npy file with the depth image (it's also zipped since github didn't like the file ending): depth.zip

Regarding the example of @saurabheights, your depth image seems to be in mm, which you account for using depth_scale=1000.0. In my case, the depth image is already in m, so to my understanding, depth_scale=1.0 should be correct. Or am I missing something there?

[saurabheights]

Yes, if its already in meter, depth_scale=1.0 is fine.

[Polymere]

I am encountering the same issue (same python and o3d versions, but on ubuntu 22.04)

[Polymere]

had a quick look at the code, and it seems that the depth_trunc parameter is never used by the PointCloud::CreateFromDepthImage function if the input depth image is in float64. Will try to fix it