Replacing Rendering components from GDExtension

[RadiantUwU]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Allowing programmers to implement their own volumetric fog (if they want to add motion to it), DLSS, replace the rendering scene loop (RendererSceneRender), modifying GI system for their own, potentially adding their own LOD system, texture streaming, extend and override behavior inside RenderingServer without requiring to override it entirely.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

• Expose RendererLightStorage, RendererMaterialStorage, RendererMeshStorage, RendererTextureStorage, RendererGI, RendererFog, RendererCanvasRender, RendererSceneRender, RendererSceneCull, RendererSceneOcclusionCull and allow overridding of them.
  • RendererSceneRender should have sdfgi renamed to hddagi internally when exposed (planned by pull/86267)
• Implementation of a RenderingServerManager
  • Allows GDExtension to replace components by themselves, substituting the ones provided by the default RenderingCompositor.
  • Move initialization code from inside DisplayServer to inside RenderingServerManager and have RenderingServerManager initialized from setup().
  • Create in project settings the Rendering Implementation tab to allow the developer to choose which components to use and override.
• Create Extension classes for the internal now exposed classes which uses virtual godot functions.

Separating them by individual components allows developers to modify specific parts of rendering, and not replace the entirety of the rendering pipeline by doing this.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

A RendererCompositor, except the elements are overridden just after DisplayServer initializes.

RenderingServerManager will override according to the tab "Rendering Implementation" inside project settings.

RenderingServerManager is created inside Main::setup() and initialized just after DisplayServer inside Main::setup2()

[!WARNING] The overriding system will fail when RendererCompositorRD is not initialized. GDExtension is limited to use RenderingDevice, or to take the vulkan instance pointer and perform its own code there at a lower level.

[!NOTE] The overriding modifications will be made to RendererCompositorRD, not RendererCompositor.

The overriding system will work as follows:

• RenderingServerManager will have an enum of overridable components, alongside functions to register components during server initialization.
• After DisplayServer initialization, RenderingServerManager will look at overridden components.
  • Where it finds default, it continues to the next.
  • Where it finds a name, it looks inside the ClassDB.
    • The old component is discarded (memdelete), and then the new one takes place by initializing from ClassDB.
  • It is repeated until it finishes going through all overridable components.
  • Initialization continues as normal.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Requires modification of the source code.

[!NOTE] This proposal has been created after feedback from the Rendering Development Meeting on 12th November.

[BrianBHuynh]

Oh this was what you were working on :0 I do think in general it would be cool to give people / GDextension devs more flexibility in what they can do due to the nature of open source and all that! I'm not too strongly opinionated other than that though and am interested in seeing what others think about it

[RadiantUwU]

Supersedes #4287

[QbieShay]

Thank you Radiant for the proposal! I think we still need a bit more input from other people of what feature they'd like integrated but can't be integrated without exposing hooks or overriding parts of the engine (DLSS is a good example)

Note that we should also question wether it's worth it, or not, to expose this via extensions. For example DLSS, there's 2 problems 1, I am making a game and i want DLSS upscaling technique

2. I am making a game with Godot, I don't want to recompile the engine, but i still want to use DLSS.

Whether this should be extensionable or not, depends a lot on how we expect people to use it. Is this something somewhat specific that can be done in a fork and impacts only a few games? Then we can probably try and reduce the maintainance cost of a fork for this (for example, make it possible to pull in a DLSS module that can be compiled with the engine, so that no files in the engine itself need to be touched for this fork) Is this something that every user in Godot that makes a 3d game may need? Then maybe it'd be cool to have a hook so that an extention developer can develop the extension, and give it back to the community, so that it can function similarly to Jolt,.

We need to answer this kind of questions before we can find a solution. I agree that it's worth to keep all of those together in a single proposal because they may share similarities, but at the same time there's a world where each of those problems requires a different, specific, solution.

[QbieShay]

Of note texture streaming is something that Godot should eventually have. Not sure why someone would implement their own LOD system, over instead trying to have improvements in the built-in one.

[Calinou]

I think focusing on specific hooks for upscaling and frame generation makes more sense, as it's much less work than trying to make the entire renderer replaceable with a GDExtension. This goes not only for us engine maintainers, but also for people implementing extensions.

There are a few reasons for this:

• Some upscalers need to be external solutions, as two major upscalers out there (DLSS and XeSS) are proprietary. Their source code is not available to compile, so you need to link against the DLL(s) they provide instead. This also applies to DLSS Frame Generation (which is a separate DLL from upscaling).
• Our support matrix (in terms of rendering drivers and methods) is starting to be pretty thorough now. In other words, there isn't much of a reason to start working on a new rendering driver or method (except for esoteric use cases where a custom build is more suited anyway, e.g. homebrew ports).
  • As @QbieShay said above, most people who miss a rendering feature in Godot would be better served by making the change in the engine source and compiling custom builds. This will only get more true as time goes on and more major rendering features are implemented. We'll surely eventually get to a scenario where only highly specialized uses will require customizing the renderer in this fashion.
• Making everything in the renderer use dynamic dispatch will likely have a performance impact in CPU-bound scenes. Many oldschool engines could get away with this (Unreal Engine 1 and Quake 2 come to mind), but it's rarely seen in modern engines where CPU utilization already gets problematic on a regular basis.

There remains the question of DLSS Ray Reconstruction which would likely need much deeper integration into the engine, while still only being available in binary form. I don't have a good answer for this, but its SDK is not publicly available yet.

[RadiantUwU]

Is this something that every user in Godot that makes a 3d game may need? Then maybe it'd be cool to have a hook so that an extension developer can develop the extension, and give it back to the community, so that it can function similarly to Jolt.

This depends on the extension developers, we won't have anything without allowing them to create it. This proposal is meant to allow implementing said hooks (aka also replacing the whole rendering engine).

We might change in a future meeting the idea of how RendererSceneRenderExtension works, allowing it to modify how rendering works by allowing GDExtension to create inject its hooks at specific times (similar to Roblox), and unlike Roblox, baking it after to make it a static allocated array of function pointers, called in order to perform the steps.

[!Note] The difference between Compositors and this new system is that it can replace existing hooks, add specifically between stages of rendering, replace opaque rendering etc.

[RadiantUwU]

Related: #7916