Depth point cloud visualization is too slow for 4k+ images

[Wumpf]

Currently, we visualizer depth maps as point clouds, each point representing a sample on the depth image.

For 4k images (3840x2160) that's 8.4mio points which means we have to do a lot of vertex & (overlapping) fragment processing. This also quickly runs into point cloud overdraw issues:

• https://github.com/rerun-io/rerun/issues/3713

Ways to address this:

Sticking with the rough renderer we have:

• downsample texture to lower resolution
  • straight forward, by far the easiest approach with pretty much guaranteed perf success
  • obvious drawback: don't see the original data
  • open questions:
    • how to expose this in the ui/api
    • how to downsample: min/max/avg? option for each? could use aggregation options as on time series?
  • bonus: this could be done dynamically depending on how far away we are from the cloud?
• look into optimizing the existing rendererer
  • we've never actually profiled it with the proper tooling (Xcode, Nsight, etc.)! What are the bottlenecks actually? Are we sure?
  • the most likely problems are unfortunately the ones that are hardest to address:
    • vertex shading overhead (the texture gets samples 6 times for each point 😱) - can we avoid this? the data per triangle has to come from somewhere, whether that's a texture or not shouldn't matter?
    • fragment overdraw: rasterization rules mandate that each fragment behaves as-if sorted by triangle index
• skip over "uninteresting" points
  • try to discard any point that is too far away or too close
  • more of a band-aid since in the worst case we still hit the same issues
  • can we keep the pipeline simple by still executing the vertex shader as before and make the discard decision in there? (by moving the triangle out of clip space)
    • likely I'd say! Profiling needed

More radical:

• raymarching
  • since it's a heightfield, we can generate max/min maps and raymarch it. There's reasons to believe this should be faster:
    • tracing rays into on a min/max field is fairly efficient in general and commonly employed for things like screen-space reflections
      • min/max map generation is trivial
      • the lower resolution levels have very high cache hit rate
    • no overdraw: right now we may have millions of points quads that may try to write themselves sorted into a single pixel
    • auto-adaptive: if the heightfield is barely visible, we have to shade less pixels whereas right now we have to process the same number of quads always which are increasingly likely to overlap creating essentially race conditions for the (fewer but still too many) remaining fragments
  • we had a prototype at some point doing it, the big draw back was that it didn't look like it should: more like a heightfield, instead of sample points everything was connected
    • it doesn't have to be like that. We can just start another ray when we hit a "cliff", thus going for the same/similar look we have right now
      • Open question: does that slow down the whole process again or is it still better?
• pre-process into continuous & non-continous areas
  • instead of rendering individual points, we draw more complex geometry
  • figuring this geometry out has to be a preprocessing step
  • drawbacks:
    • many knobs and parameters
    • more complicated rendering
    • visualization may turn out quite different

Related to

• https://github.com/rerun-io/rerun/issues/1529

[emilk]

An alternative is to find a robust solution to

• https://github.com/rerun-io/rerun/issues/3713

[Wumpf]

linked the wrong issue above initially, meant to link exactly that. Fixed now. True, once we have a general solution for large point cloud rendering we can always reduce the depth point cloud problem to a point cloud problem!