Allow irradiance and radiance to be manually sampled in `spatial` shaders

[Lauson1ex]

Describe the project you are working on

Third-person, high-fidelity visuals 3d game.

Describe the problem or limitation you are having in your project

Godot doesn't implement specular occlusions on its BDRF model, so I implemented a more complete model for my materials using bent normal maps (like Unreal, Substance, etc). I need to sample radiance and irradiance manually based on the bent normal vector instead of the normal vector. I can already sort of work around this by tricking Godot to sample it for me, by sending the bent normal vector to the NORMALMAP shader built-in, but then I had to implement my own lighting algorithms in the light shader using the actual normal vectors, otherwise lighting looks obviously wrong. You can see how this is inconvenient, plus I'm not really sampling the irradiance/radiance VoxelGIs/SHs/Reflection Probes; I'm merely tricking Godot into shading the indirectly-shaded fragments with the correct color using the bent normal vectors. Plus, I'm limited to only a single sample per fragment. Implementing the lighting functions myself is fine... mostly, since I had to only do it once, but since ATTENUATION doesn't separate distance attenuation from shadow attenuation, it introduces a lot of headaches implementing physically-correct attenuation in the light shader Note: this probably should be its own proposal.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

I propose a way to manually sample irradiance and radiance directly in the spatial shader. This way, I can simply give a normalized vector to the sampling function, and it will return me the color for that voxel's/SH's/probe's coordinate. This way I don't have to reimplement light functions myself, and simply use the diffuse_burley, specular_schlick_ggx render modes.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

I propose two interfaces for sampling irradiance and radiance:

• vec3 IRRADIANCE(vec3 normal) for sampling the nearest voxel for that fragment when using VoxelGI, the nearest Spherical Harmonic when using lightmaps, or the nearest SDF probe when using SDFGI;

• vec3 RADIANCE(vec3 normal, float lod) for sampling the environment panorama, or a ReflectionProbe, when those are defined; the lod parameter defines the blur level of the pre-filtered importance-sampled radiance, for roughness levels.

If this enhancement will not be used often, can it be worked around with a few lines of script?

I could work around this issue by using a simple microfacets shadowing algorithm for specular occlusions. This is fine for rough materials, but looks really bad for glossy reflections in my opinion and only works for Lambert diffuse lighting. Bent normals are absolutely necessary for convincing specular occlusions and the only alternative for Burley. However, there are many more applications for sampling radiance and irradiance manually, namely having more flexibility in creating light shaders, or creating stencil shadows for 3rd person platformer characters with correct environment shadow colors.

Is there a reason why this should be core and not an add-on in the asset library?

Alas, this is core.

[Calinou]

See https://github.com/godotengine/godot/pull/41415 which made it possible to implement custom radiance/irradiance in master.

cc @clayjohn

[Lauson1ex]

I see. I have seen this before. But this only allows you to override the radiance and irradiance for that fragment, but not sample them. This proposal is specifically to allow you to sample them. :)

[clayjohn]

This sort of improvement is planned for 4.x. I'd also really like to see something of this nature built into the Godot Shading Language or, alternatively, just exposing the radiance/irradiance cubemaps.

The only issue I see is that the radiance and irradiance calculations are done side by side right now (e.g. we calculate radiance and irradiance from cubemaps, then we calculate radiance and irradiance from SDFGI, etc.), this proposal would require that we separate them, which means a lot of work will be duplicated (e.g. finding the proper probe, reading from the spherical harmonic etc.)

One solution (which is planned) is to allow users to use "raw" glsl shaders that completely override the built in shader. Essentially allowing users to copy scene_forward_clustered.glsl and tweak it manually. This would give users raw access to the radiance cubemap, the SDFGI buffers, and etc.

[Lauson1ex]

The only issue I see is that the radiance and irradiance calculations are done side by side right now (e.g. we calculate radiance and irradiance from cubemaps, then we calculate radiance and irradiance from SDFGI, etc.), this proposal would require that we separate them, which means a lot of work will be duplicated (e.g. finding the proper probe, reading from the spherical harmonic etc.)

Can't this be simplified into just being able to retrieve the indirect light contribution or radiance given a vector, just like it's already done with the workaround I described, except you are returned the actual color?

[clayjohn]

Can't this be simplified into just being able to retrieve the indirect light contribution or radiance given a vector, just like it's already done with the workaround I described, except you are returned the actual color?

No. The point is, by separating indirect irradiance and indirect radiance into two separate functions, we would have to waste a lot of effort. For example, this is the code Godot uses to calculate radiance and irradiance from lightmaps when spherical harmonics are used:

uvw.z *= 4.0; //SH textures use 4 times more data
vec3 lm_light_l0 = textureLod(sampler2DArray(lightmap_textures[ofs], material_samplers[SAMPLER_LINEAR_CLAMP]), uvw + vec3(0.0, 0.0, 0.0), 0.0).rgb;
vec3 lm_light_l1n1 = textureLod(sampler2DArray(lightmap_textures[ofs], material_samplers[SAMPLER_LINEAR_CLAMP]), uvw + vec3(0.0, 0.0, 1.0), 0.0).rgb;
vec3 lm_light_l1_0 = textureLod(sampler2DArray(lightmap_textures[ofs], material_samplers[SAMPLER_LINEAR_CLAMP]), uvw + vec3(0.0, 0.0, 2.0), 0.0).rgb;
vec3 lm_light_l1p1 = textureLod(sampler2DArray(lightmap_textures[ofs], material_samplers[SAMPLER_LINEAR_CLAMP]), uvw + vec3(0.0, 0.0, 3.0), 0.0).rgb;

uint idx = instances.data[instance_index].gi_offset >> 20;
vec3 n = normalize(lightmaps.data[idx].normal_xform * normal);

ambient_light += lm_light_l0 * 0.282095f;
ambient_light += lm_light_l1n1 * 0.32573 * n.y;
ambient_light += lm_light_l1_0 * 0.32573 * n.z;
ambient_light += lm_light_l1p1 * 0.32573 * n.x;
if (metallic > 0.01) { // since the more direct bounced light is lost, we can kind of fake it with this trick
    vec3 r = reflect(normalize(-vertex), normal);
    specular_light += lm_light_l1n1 * 0.32573 * r.y;
    specular_light += lm_light_l1_0 * 0.32573 * r.z;
    specular_light += lm_light_l1p1 * 0.32573 * r.x;
}

If we separated the radiance and irradiance, we would have to sample the lightmap_texture twice as much. 4 times for irradiance and 4 times for radiance.

Of course, this is only an issue if we are talking about the radiance and irradiance from all forms of indirect light. If we are just talking about the radiance/irradiance cubemaps, then your above proposal would be much easier. As a note the current RADIANCE and IRRADIANCE built in variables only effect the radiance/irradiance cubemaps and have no impact on indirect light from other light sources.

[Lauson1ex]

Ok, I see it now.

What if, instead of two separated functions, it's just a single radiance function with a lod parameter? With lod 1.0 retrieving the most blurred equivalent of irradiance. After all the spherical harmonic is an approximation of the most blurred lod of radiance. This way, you would be sampling the lightmap_texture just 4 times. Would this simplify the implementation?

[clayjohn]

@Lauson1ex It would still require the samples to be read twice, once for each function call.

I'm sure there is a way of making it work somehow, but it might require significant rewriting of Godot's internal shader. For now, it seems like the easiest approach may be to just give users access to the raw shader, then it is up to you how you handle eveything.

[Lauson1ex]

@clayjohn For compliance with the current RADIANCE and IRRADIANCE built-ins which already works on cubemaps, I think it would be enough to sample the cubemaps them, since it's the easier option to implement, as there's currently no other way to sample radiance/irradiance cubemaps otherwise.

[Lauson1ex]

@clayjohn By the way, how will raw glsl shaders work? Just pass then to materials like regular Godot shaders and they will Just Work™?

[clayjohn]

By the way, how will raw glsl shaders work? Just pass then to materials like regular Godot shaders and they will Just Work™?

I'm not sure yet. It is an idea that we have been discussing, but we haven't figured out exactly how it will work on a technical level or how we will expose it to users.

[Calinou]

Note that horizon specular occlusion was added to master and 3.x: https://github.com/godotengine/godot/pull/51417, https://github.com/godotengine/godot/pull/51416

There are also further plans to improve specular occlusion by using the AO buffer, but this requires separating direct specular from indirect specular energy to be done more accurately: https://github.com/godotengine/godot/pull/50601, https://github.com/godotengine/godot/pull/50603