search

rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.
https://rerun.io/
Apache License 2.0
6.46k stars 316 forks source link

Pinhole projection of pointcloud doesn't work and produces cryptic error #7273

Open jleibs opened 2 months ago

jleibs commented 2 months ago

Example code:

import rerun as rr
import rerun.blueprint as rrb

rr.init("rerun_example_pinhole_perspective", spawn=True)

rr.log(
    "world/cam",
    rr.Pinhole(fov_y=0.7853982, aspect_ratio=1.7777778, camera_xyz=rr.ViewCoordinates.RUB, image_plane_distance=0.1),
)

rr.log("world/points", rr.Points3D([(0.0, 0.0, -0.5), (0.1, 0.1, -0.5), (-0.1, -0.1, -0.5)], radii=0.025))

rr.send_blueprint(rrb.Spatial2DView(origin="world/cam", contents=["world/points"]))

Results in: image

Expected: should show the points projected into the camera frame.

jleibs commented 2 months ago

It looks like there are a few broken things going on here.

The code to generate a project matrix does not look like it matches the way 2D views work in general:

https://github.com/rerun-io/rerun/blob/ef753574589dcc323d1f9f06ed7471b8467fc900/rerun_py/rerun_sdk/rerun/archetypes/pinhole_ext.py#L101-L125

In particular resolution = [aspect_ratio, 1.0] causes the image 2d bounds to be totally wrong for anything containing a real image as it will force the bounds to be 1 pixel wide.

Theoretically non-image content 3D should still project correctly, but I suspect we're invalidating some other assumption about units and scale in the renderer that causes the content to fail to project.

Lastly when the origin of the scene doesn't include a visualizer and it doesn't have any children we hit the codepath of it not being added to the view. This seems weird since the entity itself defines the bounds of the view. Introducing a 2D visualizer for Pinhole probably makes sense even if it's just something like a rectangle showing the FOV.

nikolausWest commented 1 month ago

This seems pretty serious