Compositing / effects

[LoganDark]

Like for example if I want to blur everything behind a widget (otherwise known as backdrop filter), or just scale all children of a widget equally.

The first one is more complex, the second one not quite as complex but it might still be difficult to implement. Most likely you would just render all its children into a separate texture/layer/whatever and apply effects to it before blending it onto the existing content. But, that might break hitscanning or it might cause blurriness when you scale something up, so you might have to do something a bit more exotic.

For backdrop filter, you could probably have it use its children as an alpha mask, apply the effect to the existing content, and then blend it onto said existing content using the alpha mask, if that's even possible.

Not sure how your rendering pipeline works so this may be too much complexity. Open to feedback on this.

[tronical]

I think this is a legit feature request - the ability to apply filters to sub-trees or - before that - specify that a sub-tree is to be rendered into a separate layer.

Functionality wise we already do something like that in the GL backend for clipping of rounded rectangles, but it needs to be generalised.

It's not clear yet how exactly this should be exposed in the API though.

[LoganDark]

It's not clear yet how exactly this should be exposed in the API though.

Maybe add a new builtin widget for compositing purposes, called something like Layer. Then all the compositing things can accept Layer instances as inputs, that can get interesting real quick. I don't know how you'd handle the pipelining for it though.

I think it would be really fun if we could start doing arbitrary masking and scaling and transforms this way, say use one Layer as a mask and another Layer to be masked. Though with sixtyfps's performance right now it might be infeasible, but that'll probably get better over time, especially if you switch from GL to wgpu (big if).

It will be challenging to keep things looking crisp and working as expected the entire way. For example, layers would have to know their transform before anything actually gets rendered, as that affects the mapping from physical pixels <-> logical pixels, and so would hit testing for scales and skews. Hit testing would probably have to treat wherever the layer is rendered as a portal into the layer, which could also get interesting real quick, especially with arbitrarily-shaped masks.

I'm honestly not sure how to guarantee that will never be any scaling artifacts. I would really love that property, like CSS transforms in the browser, but I'd also like the flexibility of creating slightly more complex compositions if desired. Perhaps scaling can only be done when the layer is first rendered, and you can only blit/mask/etc. layers at 1:1 scale onto the screen.

[Lucien950]

backdrop filter also very important for me. Lots of designs (like glass) need this. We have opacity? Is it a lot different?

[LoganDark]

backdrop filter also very important for me. Lots of designs (like glass) need this. We have opacity? Is it a lot different?

Opacity is a fixed-function blending mode, backdrop filters would basically require arbitrary shaders to run over the content underneath.

[Lucien950]

ok you right

[Lucien950]

or perhaps maybe not arbitrary user provided shaders but at least a good selection of backdrop filters (blur especially) would be important to me.

[LoganDark]

or perhaps maybe not arbitrary user provided shaders

I didn't say arbitrary user provided shaders. Running any shaders at all like that has a high initial cost. Once you implement any shaders at all it becomes generalizable but even that has not happened yet.

The reason for this is that right now shapes can be drawn without reading back the pixels underneath them but in order to implement blur you'd need to do that.

[kroltan]

A related but simpler request would be blending modes. Right now we only have normal overlay blending, but would be awesome to allow even just a couple blending modes like Multiply/Additive/Subtractive.

These wouldn't need intermediary rendering steps, since they just read the underlying color, and just the basic ones would be very useful and have hardware support in GPUs. For software rendering, it's already required to run blending for each pixel anyways.

My use case would be for making eye candy by overlaying elements with different blend modes to make lighting effects and similar.

Prior art:

https://developer.mozilla.org/en-US/docs/Web/CSS/mix-blend-mode

https://docs.unity3d.com/ScriptReference/Rendering.BlendOp.html