Draw order & (order independent) transparency

[Wumpf]

Current situation

Currently we do not support sorting within a Renderer, instead all renderer have a fixed order. This is primarily an issue for supporting transparency as transparent objects (with (premultiplied) alpha blending) need to be at least be ordered back to front. Note that additive blended objects are order independent but still need to be drawn after all opaque objects.

Steps needed

If (!!) we're keeping the current monolithic Render approach, we can split the way to fully supporting transparency in two steps:

• [x] allowing Render to be invoked several times for different "phases", a prototype is being developed on andreas/re_renderer/phases
• [ ] Sorting within (certain) phases either by..
  • invoking every Renderer an arbitrary amount of time with relatively fine grained draw data
  • implementing an order independent transparency technique

Pro/Con of fine grained object sorting

Sorting rendering primitives by distance to camera is very common in general purpose rendering engines! However, regardless of the other implications of an order independent transparency algorithm, not sorting within Renderer actually has both advantages and disadvantages:

Drawbacks:

• no front-to-back order for opaque, meaning potentially lot of (discarded) overdraw
  • fragment shader with discard/depth changes may not profit from ordering
  • this is an issue if pixel shading complexity is high and can be alleviated with a depth-first pass
  • profiling is needed to determine if this actually helps, may be very scene dependent. Some of our key vertex shading is complex (points & lines), making depth-first less attractive
• no back-to-front order for transparent, meaning rendering artifacts until we implement some form of order independent transparency

Advantages:

• super simple, less data to transfer & compute (don't need to know distances)
  • allows fast path from database to gpu data, close to no data examination needed
• faster in some cases
• best possible batching, we invoke every Renderer only once per pass, meaning minimal stage changes without the need for any sorting

Order Independent transparency (OIT)

Employing an OIT technique is partially independent of the question of draw order: As noted above draw order has advantages even with OIT in place but is no longer required then. Furthermore, depending on our expected scenes draw order alone will not solve alpha blending ordering issues either because our batches are not fine grained enough (i.e. a single draw has several front/back faces in line of sight) or triangles are overlapping (happens often in our case - e.g. line penetrating transparent box).

As our visualizations become more complex it seems to be unavoidable to employ at least one (maybe several for different situations or quality tiers) OIT technique. As this removes the sorting requirement, our database->renderer path could profit from being freed of the sorting limitation which is why making a directional decision here early can inform other design aspects of the renderer!

OIT techniques to evaluate

(WIP, trying to do rough families so far and do some early discarding)

• ❓ Fragment lists
  • every pixel keeps list of fragments which is later sorted and resolved
  • probably highest quality and most expensive. Comes in many variations
  • Update: Bevy now has a simple version of this which works with a fixed number of layers (rather than linked lists, this may have pros & cons as well!)
• ❓ Weighted, Blended Order-Independent Transparency
  • see also implementation guide
  • use additive blending with depth dependent weights, i.e. faking alpha blending
  • very cheap and surprisingly good results overall, but breaks down when weight heuristic (needs tweaking) fails. Can happens in particular when objects are only very slightly transparent for an example (probably with incorrect weights?) see [this implementation/blog by MJP](https://therealmjp.github.io/posts/weighted-blended-oit/
• ❓ Stochastic transparency
  • render everything opaque but interpret alpha as likelyhood for being shown
  • optionally some clever blur
  • works well together with Alpha-to-Coverage MSAA!
  • https://twitter.com/matiasgoldberg/status/1694025312785379501
• ❓Wavelet/fourier based methods
  • great recent writeup on a haar-wavelet based one here https://osor.io/OIT
  • Fourier Opacity Mapping is an older one I implemented at some point (:
  • typically great memory usage and ok'ish number of draws required (varies over concrete implementations)
  • depending on basis functions typically suffers from different kind of noise/ringing/other artifacts
• ❌ Depth Peeling
  • fixed amount of depth layers that are rendered one by one
  • rather expensive as geometry need to be rendered multiple times

[emilk]

We should consider doing something simple/stupid as a first step, something that is not order-independent. Just draw transparent objects last, for instance, just to get some of the transparency story done.

[Famok]

@Wumpf was the opacity implemented? Because the none of the opacity sliders seem to do anything, neither does setting an alpha value in RGBA values seem to have any effect.

[Wumpf]

@Famok this ticket is about implementing transparency and its prerequisites in Rerun. We have some very limited support, for example the opacity slider on images which is only really used for layering images on top of each other. If there's a bug with that please do file an issue!