rerun-io / rerun

Visualize streams of multimodal data. Fast, easy to use, and simple to integrate. Built in Rust using egui.
https://rerun.io/
Apache License 2.0
6.39k stars 312 forks source link

Automatically visualize point clouds from depth maps #1333

Closed teh-cmc closed 1 year ago

teh-cmc commented 1 year ago

Following a discussion with @Wumpf, we would like to experiment with a Renderer that is capable of raytracing through discrete volumes (i.e. an Amanatides & Woo kinda thing).

This renderer should ideally be capable of using either a 3D texture + bounding box or a 2D texture (interpreted as depth) + bounding box as input and render that in a spatial view as needed.

Our first real use-case would be to try and render depth maps in spatial views.

Today, the only way we have at our disposal to do that is to project the depth map back into 3D on the client-side, build a point cloud out of that, and upload the result to Rerun (see nyud example). Of course this is pretty costly and doesn't scale nicely with the resolution: a 4K depth map would result in a 8M+ point cloud each frame :cold_sweat:

teh-cmc commented 1 year ago

Here's a sneak peek of the first experiments on that front.

As it stands right now it is completely unusable in real conditions, not to mention the math that barely makes any sense at all... but it does show that it can look both fairly fancy and much more "readable" than a point cloud indeed!

https://user-images.githubusercontent.com/2910679/219964087-7021ef1e-8e8a-42a2-b042-7e2a79e677c8.mp4

teh-cmc commented 1 year ago

Implemented in #1421