Advanced rendering effects - filters, masks, box-shadows, gradients, and shaders

[mikke89]

This one has been a long time in the making, first mentioned in #307, now the time has come for one of the biggest additions to the library. I hope you like the changes!

I am posting this as a pull request to gather more feedback. There's a lot here, feel free to give any feedback, whether a small comment or a fully detailed review. It would be great to see the effect sample especially tested on more machines, as I'm sure there will be issues popping up. I've tried to keep the commit history clean, which should be helpful for reviewing.

There has been some feedback regarding the new render interface in particular, please let me know what you think here, before we lock-in the changes. Some thoughts here: https://github.com/mikke89/RmlUi/issues/307#issuecomment-1837466674

This PR closes #249, #253, #307, and even addresses #1.

New features

New properties:

• filter: Apply a rendering effect to the current element (including its children).
  • Supported filters: blur, drop-shadow, hue-rotate, brightness, contrast, grayscale, invert, opacity, sepia. In other words, all supported functions from CSS.
• backdrop-filter: Apply a filter to anything that is rendered below the current element.
• mask-image: Can be combined with any decorator, including images and gradients, to mask out any part of the current element (and its children) by multiplying their alpha channels.
• box-shadow: With full support for offset, blur, spread, and insets.

New decorators:

• shader: A generic decorator to pass a string to your renderer.
• linear-gradient, repeating-linear-gradient
• radial-gradient, repeating-radial-gradient
• conic-gradient, repeating-conic-gradient

The new rendering interface include support for shaders, which enable the above decorators. Parsing is done in the library, but the backend renderer is the one implementing the actual shader code.

All of the filters and gradient decorators have full support for interpolation, that is, they can be animated. This is not yet implemented for box-shadow.

Decorators can now take an extra keyword <paint-area> which is one of border-box | padding-box | content-box, which indicates which area of the element the decorator should apply to. All built-in decorators are modifed to support this property.

Custom filters can be created by users by deriving from Filter and FilterInstancer, very much like how custom decorators are created.

Improved element clipping behavior. Handles more complicated cases, including nested transforms with hidden overflow, and clips to the curved edge of elements with border-radius. This requires clip mask support in the renderer.

New effect sample showcasing a lot of the new features.

Major overhaul of the render interface

The render interface has been simplified to ease implementation of basic rendering functionality, while extended to enable the new advanced rendering effects. The new effects are fully opt-in, and can be enabled iteratively to support the features that are most desired for your project.

Highlighted changes:

• Now using safer, modern types (like span).
• A clear separation between required functions for basic functionality, and optional features for advanced rendering effects. The required functions are now pure virtual.
• All colors are now submitted as 8-bit sRGBA (like before), but with premultiplied alpha (new). Existing renderers should modify their blending modes accordingly. This change is vital for correct blending of partially transparent layers.
• All geometry is now compiled before it can be rendered, which helps simplifying the interface.
  • Now the pointers to the geometry data (vertices and indices) are guaranteed to be available and immutable until the same geometry is released. Thus, users can simply store the views to this data, and reuse that during rendering, which should help considerably with migrating from the immediate rendering function.
• The scissor region should now be applied separately from any active transform. Previously, we would have to manually redirect the scissor to a stencil operation, that is no longer the case. Instead, the clipping with transform is now handled by the library, and directed to the clip mask functionality of the render interface as appropriate.
• Expanded functionality to enable the new rendering effects, including layered rendering, rendering to texture, rendering with filters and shaders.
• Textures are no longer part of the compiled geometry, compiled geometry can be rendered with different textures or shaders.

Backward compatible render interface adapter

The render interface changes will require updates for all users writing their own render interface implementation. To smooth the transition, there is a fully backward-compatible adapter for old render interfaces, see RenderInterfaceCompatibility.

1. In your legacy RenderInterface implementation, derive from Rml::RenderInterfaceCompatibility instead of Rml::RenderInterface.

      #include <RmlUi/Core/RenderInterfaceCompatibility.h>
      class MyRenderInterface : public Rml::RenderInterfaceCompatibility { ... };

2. Use the adapted interface when setting the RmlUi render interface.

      Rml::SetRenderInterface(my_render_interface.GetAdaptedInterface());

It can also be useful to take a closer look at the adapter before migrating your own renderer to the new interface, to see which changes are necessary. Naturally, this adapter won't support any of the new rendering features.

Render manager and resources

A new RenderManager is introduced to manage resources and other rendering state. Users don't normally have to interact with this, but for contributors, and for more advanced usages, such as custom decorators, this implies several changes.

The RenderManager and can be considered a wrapper around the render interface. All internal calls to the render interface should now go through this class.

Resources from the render interface are now wrapped as unique render resources, which are move-only types that automatically cleans up after themselves when they go out of scope. This considerably helps resource management. This also implies changes to many central rendering types.

• Mesh: A new type holding indices and vertices. Can be constructed directly or from MeshUtilities (previously GeometryUtilities).
• Geometry: Is now a unique resource holding a compiled geometry handle, and constructed from a Mesh, taking ownership of the mesh's data.
• Texture: Now simply a non-owning view and can be freely copied. The underlying file texture is owned by the render manager, and held throughout the manager's lifetime.
• CallbackTexture on the other hand is a unique render resource, automatically released when out of scope.

See the following commit message for more details: a452f26951f9450d484496cccdfad9c94b3fd294.

Other changes

Utilities:

• Improved mesh utilities to construct background geometry for any given area of the element, including for elements with border-radius.
• New Rectangle type to simplify many operations, transmitted as scissor region in render interface.
• Debugger now displays the axis-aligned bounding box of selected elements (including any transforms and box shadows).

General improvements:

• Use RCSS syntax for color-string conversion, format colors using hexadecimal notation.
• Fix some situations where units were not shown in properties, ensure all invoked types define a string converter.
• Use default log output also when there is no system interface installed, and redirect all print-like calls to the built-in logger. This ensures that log messages are submitted to the same stream output before and after installing the default provided system interface. In particular, the output from MSVC is given in its debug output.

Fixes:

• In demo sample, fix form submit animation not playing smoothly on power saving mode.

Visual tests:

• Several new visual tests for the new features.
• Highlight differences when comparing to previous capture by holding shift key.

Limitations

Filters will only render based on geometry that is visible on-screen. Thus, some filters may be cut-off. As an example, an element that is partly clipped with a drop-shadow may have its drop-shadow also clipped, even if it is fully visible. On the other hand, box shadows should always be rendered properly, as they are rendered off-screen and stored in a texture.

Breaking changes

• Render interface.
  • Signature changes to several functions, see notes above.
  • Texture is no longer part of the CompileGeometry command, instead it is submitted with the Render... command(s).
  • Implementing the new clip mask API is required for handling clipping of transfomed elements.
    • Note that, this should require very trivial changes now that the vertex and index pointers are guaranteed to be valid until ReleaseCompiledGeometry is called.
  • See RenderInterfaceCompatibility notes above for an adapter from the old render interface.
  • RmlUi now provides vertex colors and generated texture data in premultiplied alpha.
    • Set the blend mode to handle them accordingly. Recommended to convert your own textures to use premultiplied alpha. This ensures correct compositing, especially with layers and effects.
  • Font effect color output now assumes premultiplied alpha. Color channels are initialized to black (previously white).
• RenderManager should now be used for all rendering instead of manually calling the render interface.
• Redefine Geometry, introduce Mesh.
  • Mesh is used to define vertices and indices, before it is submitted to construct a Geometry through the render manager.
  • Geometry is now a wrapper around the underlying geometry (to be) submitted to a render interface.
    • Move-only type which automatically releases its underlying resource when out of scope.
  • GeometryUtilities (class and header file) renamed to MeshUtilities.
    • Signatures changed to operate safely on a mesh instead of using raw pointers.
  • Geometry no longer stores a Texture, it must be submitted during rendering.
• Redefine Texture.
  • This class is now simply a non-owning view of either a file texture or a callback texture.
  • CallbackTexture is a uniquely owned wrapper around such a texture.
  • These are constructed throught the render manager.
  • File textures are owned by and retained by the render manager throughout its lifetime, released during its destruction.
• Font engine interface.
  • GenerateString signature change.
• Decorator interface: GenerateElementData has a new paint area parameter.
• Moved DecoratorInstancer into the Decorator files.
• The old gradient decorator has been deprecated, instead one can now use horizontal-gradient and vertical-gradient, thereby replacing the keyword to indicate direction.

Not yet implemented

Some features to be implemented, or being considered:

• Only OpenGL 3 renderer implements all new rendering features for now, all other backends have been updated to work with the updated render interface but with their old feature set.
• The blur algorithm could need more research / tweaking. I've tried to make it relatively performant also for larger blur radii, but it needs proper performance studies (benchmarks) and more rigorous quality tests. I'd be interested if someone wants to dive into that.
• Consider enabling decorators in the background property. This way, we can use e.g. background: linear-gradient(...) like in CSS.
• Animated box-shadow: Render live instead of generating texture.
• FontEngineInterface: Change ptr,size to Span.
• Effect sample has graphical glitches when built with emscripten.

I consider none of these blocker, instead we can do them as follow-ups.

Screenshots

I'll close off with some eye candy:

Effect sample

[0suddenly0]

It seems to me that there are problems with the new FileTextureDatabase, because after unloading all textures, only font textures are reloaded, this can also be seen in tests.

After calling Rml::ReleaseTextures(), counters.release_texture == 4, because 3 generated textures are unloaded, and one loaded.

But after passing Update+Render, only the number of generated textures increases, the number of loaded textures remains the same.

You can also simply release all the textures in any example, after which only the font textures will be loaded again.

[mikke89]

Good catch! Thanks for reporting.

I made some changes, I believe this one should fix this issue: 011b3c17efeaff5a6bcda4547dd7fd4cb5590451 Extended some tests too so we'll catch this next time.

[0suddenly0]

Due to the installation of scissors within the boundaries of the element, when drawing any filters, the child elements that are behind the object are cut off, despite the overflow properties.

The blue element is a child element of the red one. filter in chrome:

the same filter in rmlui:

[mikke89]

I see, I'll look into that. Can you post a full document example?

[0suddenly0]

I see, I'll look into that. Can you post a full document example?

<html>
    <head>
        <style>
            div { position: absolute; }
            div.parent {
                top: 50px;
                left: 50px;

                width: 40px;
                height: 40px;

                background-color: red;

                filter: blur(10px);
            }

            div.child {
                top: 100px;
                left: 100px;

                width: 100px;
                height: 100px;

                background-color: blue;
            }
        </style>
    </head>
    <body>
        <div class="parent">
            <div class="child"></div>
        </div>
    </body>
</html>

[0suddenly0]

There are also some problems with applying opacity and filter properties to the backdrop-filter. If we use such code, we will see that RmlUi does not impose sepia on the background:

<html>
    <head>
        <style>
            div { position: absolute; }
            div.bg {
                top: 75px;
                left: 75px;

                width: 50px;
                height: 50px;

                background-color: green;
            }

            div.fg {
                width: 200px;
                height: 200px;

                backdrop-filter: blur(10px);
                filter: sepia(1);
            }
        </style>
    </head>
    <body>
        <div class="bg"></div>
        <div class="fg"></div>
    </body>
</html>

chrome

RmlUi

For Chrome, this is correct only if the div.fg there is no backdrop-filter.

[mikke89]

@0suddenly0 I finally got around to looking into these ones.

For the first issue with a child and filter parent, this is a bit tricky for us. It has to do with how absolutely positioned elements are considered in terms of overflow. Specifically, a known difference from CSS is that we don't show scroll bars or clip such elements when placed outside their offset parent's visible region. There is some previous discussion on this topic, but in short, we want absolute positioning to be independent of layout, so that we don't have to compute an expensive layout step every time we move something around the screen.

This behavior extends to filtering, such that absolutely positioned elements are not considered a part the current element's visible region. I am considering a way to make this work without inducing a full re-layout step. In particular, we could keep track of any such absolutely positioned overflowing elements, and say that:

1. Positioning such elements still don't affect layout and specifically don't make scrollbars appear.
2. However, they will affect the visual clipping region of their ancestors, even if they don't affect visibility of scrollbars.

This could solve some existing cases where one would need to use clip: always. However, it requires some considerations and more work, we'll possibly need elements to keep track of their absolutely positioned descendants. For now it is considered outside the scope of this PR, we'll have to revisit this later. I made a visual test case for when we revisit this.

---

I've pushed a commit for the second case. There was indeed a limitation to combining backdrop-filter with filter and mask-image, so thank you for reporting this. I made some changes, including a new test case for several different combinations of these properties:

I also identified some cases where the edges of backdrop-filter with blur would be dark, as seen above. There might still be some cases left to consider for these combinations. The resulting code is a bit clunky in my opinion, so I am consider making some render interface API changes around layers.

Also, please note that our opacity property works a bit differently from CSS, but filter: opacity() should generally work the same as in web browser.

[wh1t3lord]

@mikke89 can you add a such sample to official samples please? Or how to call a such sample? It might be useful for testing render backends since we would have many of them...

[mikke89]

@wh1t3lord All of these examples are added to our visual tests. You can find them by launching the VisualTests target included with BUILD_TESTING=ON.

The jellyfish in the screenshot above was replaced with an existing image in the repository for the test though. I have a jsfiddle here for the browser equivalent for anyone interested.