Godot 4 Ultimate Flexible Outline and X-Ray Shader

[alfredbaudisch]

Godot 4 lacks proper Outline and X-Ray/Silhouette shader solutions, meanwhile Unity and Unreal Engine provide multiple professional solutions for outlines. All current Godot 3 samples and open-source projects work to a certain degree, but all of them have flaws and/or are broken in a way or another with Godot 4.

In the end, neither of the existing Godot 3's solutions would be currently usable in games that require Outlines and X-Rays, i.e. the majority of 3D games nowadays use either Outlines and/or X-ray.

The current solutions either break the mesh existing material or create depth and culling problems, especially with animated skeletal meshes. The link of solutions tested are at the bottom.

This image is taken from a Unity project using Quick Outline.

Objective

The aim of the project Godot 4 Ultimate Flexible Outline and X-Ray Shader is to provide a flexible solution for:

• Toggeable outlines at runtime
• Multiple outline colors customizable at runtime
• X-ray for occluded parts of the mesh
• Support for skeletal or static meshes, animated or static
• Support for any underlying material/shader

See each feature in more details below, with sample images/videos.

The samples below were produced in Unity using a free solution called Quick Outline (open-source). The idea is to replicate Quick Outline for Godot.

But in our case, it doesn't matter whether it's made through post-processing, multiple material passes, cull masks, etc. The only requirement is that it must have the features listed below. Unfortunately Godot still doesn't have stencil buffers - which requires creativity and hacky ways to implement this.

Open-Source and Paid Work

Everything implemented will be open-source, available for anyone to use, and to help strengthen Godot - especially considering that this is a MUST for the majority of 3D games nowadays.

This way, any help is appreciated. And if you want to do this, but as paid work, feel free to let me know. I have a personal budget that I am willing to invest in order to sponsor this project.

If you are interested, you can open a PR, leave a comment this issue or send me an email: https://alfredbaudisch.com/contact

Features

Customizable Colors At Runtime

This image is taken from a Unity project using Quick Outline.

Toggeable At Runtime

https://user-images.githubusercontent.com/248383/198830529-32a4ebd3-6eac-4054-baa8-eab1c54b6716.mp4

This video is taken from a Unity project using Quick Outline.

Respect the Mesh's Existing/Underlying Material

The solution should support any underlying material. For example, if the skeletal mesh has its own Material, it must be possible to still use it. It doesn't matter if the material appears in the first or pass for example, as long the existing mesh material can be used, no matter whether it's a default PBR Material or a custom material with a custom shader.

It's also OK to make adjustments to the material overrides or overlays, the important thing is that the mesh's original visuals are still shown.

For example, this character material work as is, even with the outline:

This image is taken from a Unity project using Quick Outline.

Support for Static Meshes and Skeletal Animated Meshes

All previous examples show the characters walking on purpose, highlighting that the outline works with skinned meshes and deformations.

Occlusion XRay/Silhouette

Preferable there should be an option to choose between Outline + Silhouette, just the Outline or just the Silhouette when occluded. But to make things simpler, showing only the Silhouette when occluded is more than enough.

https://user-images.githubusercontent.com/248383/198830661-99fb679c-e438-4f9d-8d75-9be7acff1816.mp4

This video is taken from a Unity project using Quick Outline.

Base Project

A Godot 4-beta3 base project is provided with the same characters from the Unity samples below. The main goal with this project is to replicate the features of Unity's Quick Outline with Godot 4 in the Base Project: GodotUltimateOutline-BaseProject.zip

References

Godot 3 Outline Projects

• https://github.com/Reisflocke/godot-3d-outline
• https://github.com/ualac/godot-outline3d
• https://github.com/GDQuest/godot-shaders
• https://godotshaders.com/shader/pixel-perfect-outline-shader/
• https://www.youtube.com/watch?v=9JuZV6Xeo-w#2

Unity

• Quick Outline (repository)

[Calinou]

Having outline support will be useful to implement https://github.com/godotengine/godot-proposals/issues/2795.

For drawing outlines, the built-in approach could work much better for meshes that don't use smooth vertices exclusively if Godot could generate better outline meshes. This can already be done manually in 3D authoring software, but it requires exporting a separate copy of the mesh for use as an outline.

An X-ray shader will likely require https://github.com/godotengine/godot-proposals/issues/1298 to be implemented to be done properly.

PS: Remember that for GitHub video previews to work, you need to have a blank line before and after the video URL.

[alfredbaudisch]

Having outline support will be useful to implement godotengine/godot-proposals#2795.

Currently, I am looking for an outline solution that doesn't require changing the engine itself - an implementation with the current features. But of course that would welcome as well.

An X-ray shader will likely require https://github.com/godotengine/godot-proposals/issues/1298 to be implemented to be done properly.

I already managed to create a X-Ray with depth_test_disabled then adding another pass with higher priority with the mesh's material, but it works only with depth_draw_never, which leads to wrong rendering of the mesh when not occluded.

PS: Remember that for GitHub video previews to work, you need to have a blank line before and after the video URL.

Thanks, fixed.

[alfredbaudisch]

@Calinou In the end there's really no way of doing it currently with what we have in Godot 4? Even if many approaches are mixed?

[Zireael07]

I saw an x-ray shader in I think GDQuest's collection you linked, and I know I saw a solution somewhere but that one involves a freaking stack of 3+ viewports... was unusable in practice.

[alfredbaudisch]

I saw an x-ray shader in I think GDQuest's collection you linked

Unfortunately it has all the flaws that need to be solved that I linked in the Issue.

one involves a freaking stack of 3+ viewports

I also tested this one with viewports. It provides no way of changing color in runtime unless you create 3 more viewports for each color.

And regardless, it's buggy in Godot 4, it creates artifacts.

[Calinou]

@Calinou In the end there's really no way of doing it currently with what we have in Godot 4? Even if many approaches are mixed?

No, not without modifying the engine source code to implement https://github.com/godotengine/godot-proposals/issues/1298. It's probably not very difficult, but Vulkan exposes this differently compared to OpenGL.

In the meantime, if you want selected characters' positions to be visible through walls, you can use a Sprite3D with a "ring" texture around the character's feet that is set to be visible through walls. This is similar to what many RTS games do when selecting units.

[apples]

Howdy! I've been exploring this road, I hope y'all don't mind if I add some notes here.

I've experimented with various outline/silhouette shaders, without stencil support, using https://github.com/godotengine/godot/pull/73527 and some solutions to https://github.com/godotengine/godot/issues/73158.

With those solutions in place, it's possible to make a nice silhouette shader for a single character in the opaque pass. However, depth prepass cannot be used with this type of shader, and there are some other small issues especially with overlapping silhouettes.

Viewport-based solutions are entirely unwieldy and basically are not an option for typical games.

It seems to me that stencil buffer support is absolutely necessary to get a good silhouette that doesn't have these problems.

To me, the ideal scenario would be using a material_overlay on the geometry to write stencil values where depth passes an equal check, and then a next_pass material which renders the silhouette only outside of the stencil region. Perhaps a new StencilMaterial3D type could be created to make this easy to do without bloating the existing BaseMaterial3D.

That is how the Unity Quick Outline addon is designed, and it is very effective with minimal issues. I've also been observing how other games draw silhouettes, and this approach seems common.

That is the design I will be working on in my personal endeavors.

[alfredbaudisch]

Thanks for the contribution @apples !

It seems to me that stencil buffer support is absolutely necessary to get a good silhouette that doesn't have these problems.

100%.

[apples]

Progress update:

Godot 4.0 seems to have a working stencil buffer, but stencil operations are currently not exposed through the Material APIs.

To implement this, I had to add the relevant fields to StandardMaterial3D.

These fields are reproduced in the shader using render modes I added:

shader_type spatial;
render_mode stencil_compare_always,
            stencil_pass_replace,
            stencil_depthfail_replace,
            stencil_reference 1,
            stencil_comparemask 255,
            stencil_writemask 255;

A bit wordy, and requires a (surprisingly small) parser modification to allow for the numeric parameters, but it gets the job done.

When combined with https://github.com/godotengine/godot/pull/73527, I can achieve the results shown above.

Note: It seems like all the necessary APIs are exposed via RenderingDevice, but I'm not savvy enough with that API to implement anything using it. But it seems like this might be able to implement stencil effects entirely from script?

My current working branch can be found here: https://github.com/apples/godot/tree/depth-function+stencil-buffer

[anaanook]

@apples this is great! Tested out your branch and it works well. This feature should definitely be added officially!

[alfredbaudisch]

@apples thanks a lot for this, this is great!

[jbromberg]

Looks like stencil support will be merged soon: https://github.com/godotengine/godot/pull/80710

[andersmmg]

I think it's still on hold, the code is approved but I don't think they want to merge it until the render priority issue is figured out

[permelin]

I already managed to create a X-Ray with depth_test_disabled then adding another pass with higher priority with the mesh's material, but it works only with depth_draw_never, which leads to wrong rendering of the mesh when not occluded.

I'm probably missing something obvious here but I've done the same thing combined with checking the depth texture to see if we're occluded, and if not just set alpha to zero. It seems to work but I haven't tested it extensively.

Only problem I've come across so far is how to specify objects that actually always should occlude, for example enemies. That can be worked around by setting enemy material transparency to depth pre-pass.

shader_type spatial;
render_mode depth_draw_never, depth_test_disabled, unshaded;

// This shader draws a semi-transparent single-color overlay when the object is
// occluded. When not occluded, ALPHA is set to zero. Use this in a next_pass
// material.

uniform vec3 overlay = vec3(0.3, 0.3, 0.9);
uniform float opacity = 0.5;
uniform float bias = 0.1;

uniform sampler2D depth_texture: source_color, hint_depth_texture;

void fragment() {
    // Convert depth texture coordinate from screen space to clip space to
    // view space
    float clip_depth = texture(depth_texture, SCREEN_UV).r;
    vec3 clip_space = vec3(SCREEN_UV * 2.0 - 1.0, clip_depth);
    vec4 view_space = INV_PROJECTION_MATRIX * vec4(clip_space, 1.0);
    view_space.xzy /= view_space.w;

    // Set non-zero alpha if we are further away from the camera than whatever
    // is in the depth texture
    ALPHA = opacity * float(-view_space.z + bias < -VERTEX.z);
    ALBEDO = overlay;
}