Shader pipeline rework for more flexibility

[MegadronA03]

Describe the project you are working on

I'm exeprimenting with different aspects of the engine.

Describe the problem or limitation you are having in your project

Currently facing troubles in portal, skybox, pre and post effect implementations. They are implemented not in the most flexible and convinient ways.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

SubViewport require quite a lot of VRAM, for simple features like drawing multiple skyboxes in the scene, or drawing UI without texture restrictions, and this also will allow to eliminate useless fragment shader executions in rasterization process.

The new system not only will make rendering pipeline much more transparent and flexible to the end user, but also allow much easier material modification and fast post processing, without problems with transparency.

This change will allow to modify node rendering properities, and add like transparency, or cutting rendered meshes using masks or discard shader. That also solves the problem with transparency, because post processing literally modify the end material of the object, instead of actually doing 1 more layer of overdraw.

this change also allow (like in source games) to have any number of skyboxes in currently active camera (tho only one will contribute to camera lighting).

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I think its better to approach shader typing like godot does it with its objects (like we have some base empty shader class, that have Sky, CanvasItem/2D, 3D, Prep, Post childern that are extending(inhereting uniforms, functions and variables) from it with their own uniform hints and also could be converted/extended into your custom shader class for editing, just like objects in godot)

the base hierarcy that I thought of:

Shader (base resource for the shaders)
    Prep (pre depth processing)
    Post (post processting, expects color as input)
        Environment (just has compilation of multiple shaders used for the same things as in Environment resource)
        LightSettings (ambient and reflected light modifiers)
        Tonemap
            Linear
            Reinhard
            Filmic
            ACES
        Adjustments (contrast, brightness, saturation and color correction, tho I think to split it into different classes)
        Screen Read (this shader reads from provided textures, that should be screen texture, but not limited to it)
            SSR
            SSAO
            SSIL
            Bloom/Glow
                FFT based
                Blur based (idk how its called)
            Depfh Effects
                DOF Blur
                FogLinear (literally the cheapest fog that only rely on depth value)
                FogRadial (radial fog, without complex shapes)
            Volumetric (used for smoke or advanced volumetric fog)
    CanvasItem/2D (camera and node transforms)
    Sky (camera basis or something)
        Box
        Physical
        Procedural
        Cylinder
        Panorama
        Layered (for using skyboxes with transparency, so part of the skybox could be changed)
    Backbuffer (used for Screen Reading shaders)
    Frame (used for Screen Reading shaders)
    Color
        Unshaded
            StandartMaterial (the one with different material rendering stuff)
    3D (camera and node transforms)
        MultiMesh
            GPUParticle

956 (will allow to modify 2D transforms of 3D projections on screen)

Each shader will have "next pass" and "prep pass" that allow to construct desired shader using other shaders. this could be more flexible in terms of adding obscure functionality, like drawing 2D elements directly In 3D or the other way around (3D on 3D projection) without limitation of the SubViewport node.

4564

WorldEnvironment should be changed to WorldLight or LightSettings/LightEngine (because suggested implementation literally deconstructed Environment resource into multiple seperate post processing shaders), that controls settings of the light engine (like SDFGI and one skybox shader that contributes ambient and reflected lighting).

2196

for post processing I think its better to add 1 node:

- for modifying the way material is drawn to the screen (post processing effects (it could be used for volumetric editing) and pre depth processing + fragment discarding) for cutting of parts of rendered image using masks or something other (like cutting mesh using plane + discard) and not wasting performance on useless fragment calls. Also with setting to ignore mesh rendering mask (fullscreen effect flag), so we could have bloom (aka glow) without cutouts from the mesh

post processing shader resource (if there are no plans on modifying the rendering pipeline):

- add new screen texture hint, when all nodes are rendered except those with that post processing
- if this shader type doesnt relies on data from screen (screen hint textures and etc), it could be optimized for low end devices, providing effects without actually drawing triangle that covers the screen with another fragment pass.
- Note that varying(or idk if there will be other keyword for that) variables should be avalable in next shader pass. (like passing variable from prepass to material, from material to post process)

This node will "add" another next pass shader to each child nodes (or to specific nodes with its parents (might be specified in the node itself) (subject to change/review)), so it should be much more flexible in terms of applying different effects for multiple things on the screen.

if currently suggested idea breaks some feature of godot, feel free to write about that and suggest a solution.

If this enhancement will not be used often, can it be worked around with a few lines of script?

As I said, yes, but at the cost of VRAM and GPU performance, or will require very messy implementation of modifying multiple next pass shaders(and thats only for the Spatial materials).

Is there a reason why this should be core and not an add-on in the asset library?

This changes the root of a problem (render pipeline), so add-on or asset library would not be enough for this.

[MegadronA03]

P.S. thought that current suggestion needs new proposal, because it solves multiple issues at once

[jams3223]

That's a very good idea actually, it keeps it clean.

[MegadronA03]

Also according to what I said, camera also needs some new features, like content clipping and custom projection transformstion #2713, so the camera view could be freely transformed relative to the screen and other systems like LOD or occlusion culling could work with this.

[MegadronA03]

elaborating on previous message: About cameras (3D, 2D), looks like its should have its clipping container (CameraContainer, that defines aspect ratio, and its scaling based on its size, tho to put camera into container, it just should be specified, instead of adding it as a child, because that would break some character controllers), if there no CameraContainer specified, it will render just with own 2D transform. Camera will always render projection if its visible in a tree, so multiple cameras could be used with different 2D transformations relative to the screen (yes it's makes make_current() obsolete). Also take out of SubViewport node functionalitiy for creating new World3D and put that into a seperate node that could be specified (using parent child system in a scene tree) in camera and SubViewport for rendering multiple different scenes, without using/spending visibility layers.

Edit: so CameraContainer will apply UI transforms to the 2D transform of the camera(2D,3D) 2D(screen) projection.

Camera will need this change, because if this proposal will be implemented, the shaders will also able to change transforms of meshes more extensively than traditional cameras.