Point cloud is sometimes incomplete near camera (ZED2i)

[Victorlouisdg]

Describe the bug Very rarely when I start using a ZED 2i camera, a part of the point cloud close to the camera is missing. Screenshot 1 shows the incomplete point cloud, screenshot 2 shows the point cloud after restarting my script. For me this happened when running the ZED2i at 2K resolution 15 fps with neural depth.

I suspect the problem is that the depth map contains invalid values for the pixel with points close to the camera.

Screenshots

Environment:

• Python: 3.10
• ZED SDK 4.0

[tlpss]

and would this be something on our side or an issue in the ZED SDK?

[Victorlouisdg]

I forgot to mention: I'm using the point_clouds branch or airo-mono, but I believe the issue can also happen on the main branch. I suspect this to be an issue with the ZED SDK because it goes away after reopening the camera, and we don't do anything special there.

[tlpss]

@Victorlouisdg should we close this?

[Victorlouisdg]

This is not resolved yet. It is a different issue than the incomplete point clouds solved in PR #100. However, I haven't noticed this issue again since I opened it, and can't reproduce it. So I guess we can close this unless it happens again.

[Victorlouisdg]

I had the same issue happen again today. I've opened the Zed2icamera at least 10 times without problems, and then suddenly it gives incomplete point clouds. This is resolved by opening the camera again.

It looks like all values with depth > 1.0 are missing:

[Victorlouisdg]

This keeps happening sporadically to me, maybe ~5/100 times I've opened the Zed camera in the last 2 days. I've written this snippet to try and catch it and at least stop my program at start-up instead of failing/getting stuck somewhere in the middle:

camera = Zed2i(resolution=Zed2i.RESOLUTION_2K, depth_mode=Zed2i.NEURAL_DEPTH_MODE, fps=15)

def check_zed_point_cloud_completeness(camera: Zed2i):
    # Check whether the point cloud is complete, i.e. if there are any points closers than 1.0 meters
    point_cloud = camera.get_colored_point_cloud()
    image_rgb = camera.get_rgb_image_as_int()
    confidence_map = camera._retrieve_confidence_map()
    depth_map = camera._retrieve_depth_map()
    depth_image = camera._retrieve_depth_image()

    distances = np.linalg.norm(point_cloud.points, axis=1)
    print(distances.shape)
    if not np.any(distances < 1.2):
        print(distances)
        logger.info("The point cloud is not complete, logging it to rerun.")
        import rerun as rr

        rr.init("Competition Station - Point cloud", spawn=True)
        rr.log("world/point_cloud", rr.Points3D(positions=point_cloud.points, colors=point_cloud.colors))
        rr.log("image", rr.Image(image_rgb).compress(jpeg_quality=90))
        rr.log("depth_image", rr.Image(depth_image).compress(jpeg_quality=90))
        rr.log("depth_map", rr.DepthImage(depth_map))
        rr.log("confidence_map", rr.Image(confidence_map))

        raise RuntimeError("The point cloud is incomplete. Restart the ZED2i camera.")

A normal log to rerun should look like this:

[adverley]

Can you try to reproduce this with only the ZED SDK? Because then we can escalate it to their side.

[Victorlouisdg]

I've had it happen again, confidence map seems ok, but depth image is affected and depth map and point cloud are even missing in Rerun:

[Victorlouisdg]

Some relevant logs:

[inf inf inf ... inf inf inf]
2024-02-08 09:03:15.334 | INFO     | cloth_tools.stations.competition_station:check_zed_point_cloud_completeness:44 - The point cloud is not complete, logging it to rerun.
Found existing process on port 9876. Trying to connect.
Traceback (most recent call last):
  File "/home/victor/cloth-competition/scripts/data_collection/05_grasp_highest_lowest_loop.py", line 10, in <module>
    station = CompetitionStation()
  File "/home/victor/cloth-competition/cloth-tools/cloth_tools/stations/competition_station.py", line 68, in __init__
    check_zed_point_cloud_completeness(camera)
  File "/home/victor/cloth-competition/cloth-tools/cloth_tools/stations/competition_station.py", line 54, in check_zed_point_cloud_completeness
    raise RuntimeError("The point cloud is incomplete. Restart the ZED2i camera.")
RuntimeError: The point cloud is incomplete. Restart the ZED2i camera.
[2024-02-08T08:03:16Z ERROR re_space_view_spatial::parts::images] Failed to create texture for "depth_map": Tensor has non-finite values

The problem seems to be that the depth map contains inf for almost everything closer than 1.2 m to the camera. It could be a problem with the NEURAL depth mode.

[Victorlouisdg]

It's happening a lot this morning, about ~1/3 times. The topdown view of the point cloud is really suspicious:

[m-decoster]

I can confirm that this is also happening with the Zed2i camera I am using, I've had it happen a couple of times today and earlier this week. Also 2k, 15 fps, neural depth mode.

[Victorlouisdg]

This is still an issue on the new desktop with a fresh Ubuntu 22.04 and ZED SDK install.

As Mathieu also has this issue with a different camera, I think we can conclude this is a bug in the ZED SDK (possibily only with NEURAL depth mode).

[adverley]

Can we raise an issue with stereolabs for this?

On Mon, Mar 25, 2024, 2:34 PM Victor-Louis De Gusseme < @.***> wrote:

This is still an issue on the new desktop with a fresh Ubuntu 22.04 and ZED SDK install.

As Mathieu also has this issue with a different camera, I think we can conclude this is a bug in the ZED SDK (possibily only with NEURAL depth mode).

image.png (view on web) https://github.com/airo-ugent/airo-mono/assets/23340946/ff45b4e0-7c76-4ff6-bc40-be420ab14342

— Reply to this email directly, view it on GitHub https://github.com/airo-ugent/airo-mono/issues/104#issuecomment-2018019893, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABLT5VBAWKYB72VVEIGXCXDY2ARVBAVCNFSM6AAAAABBLB4OVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJYGAYTSOBZGM . You are receiving this because you commented.Message ID: @.***>

[m-decoster]

Probably already known, but just confirming:

• This happens the first time I ask for a point cloud
• If I keep asking for a point cloud in the same running script instance, it keeps happening
• So, to get a complete point cloud we need to reopen the camera

Since we are opening the camera according to the SDK (afaik), this does seem like an issue for stereolabs