IBL shadowing

[MiiBond]

This PR adds an IblShadowsRenderPipeline that adds shadowing for image-based lighting (IBL). It's applied as a post process after rendering your scene. You must assign an HDR environment with setIblTexture to enable the shadowing. Both cubemaps and equirectangular projected IBL's work.

This pipeline builds a voxel grid for all the meshes in the scene that you want to cast shadows. You can exclude meshes (e.g. the skybox, ground plane, etc.) by calling addExcludedMesh on the pipeline. The voxel grid needs to be re-generated every time something in the scene moves and this can be an expensive operation (depending on the voxel resolution) so it's most suitable for static scenes in its current form. Note that the camera can move and the IBL environment is free to rotate or change completely without having to rebuild the voxel grid.

Using this voxel grid, shadows are generated by sampling the IBL and then applied to the scene as a greyscale approximation. In combination with this, we can also calculate screen-space shadows to handle shadowing of small details.

Some important exposed settings:

• voxel resolution - the dimension of the voxel grid, in voxels. Directly affects how sharp the shadows are. Must be power-of-two.
• sample directions - how many samples of the voxel grid per pixel. Increasing this results in less noise but adds significantly to the per-frame cost of shadows
• toggling screen-space shadows - SS shadows add small shadow detail close to the shadow caster but may not be needed in all scenes so this is togglable and has various settings to control how they behave.

Updating the voxel grid:

• When a mesh is added or removed from a scene, the voxelization of the scene is automatically rebuilt.
• When a mesh moves in the scene, you must call updateSceneBounds to get the shadows to rebuild.

Things that still need work:

• The voxel tracing is currently done in an extremely naïve way and tends to miss voxels, especially at higher voxel resolutions. I have an algorithm using octrees mostly implemented that will be significantly better quality but I wanted to get this PR in sooner.
• The shadows are greyscale and are an approximation of the light intensity blocked by an object. They often darken even unshadowed surfaces more than they should. I have plans to improve this and also add better support for coloured lighting.
• IBL shadows don't currently work with multiple cameras (mostly due to this issue https://forum.babylonjs.com/t/viewport-issue-when-using-prepassrenderer/51276)
• There are sometimes graphical artifacts when moving the camera. I believe that these may be due to precision in the motion buffer (the prepass "velocity" texture).
• WebGPU is not yet fully supported.
• Various performance and quality tweaks. e.g. support downsizing the buffers for faster speed.

[bjsplat]

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s). To prevent this PR from going to the changelog marked it with the "skip changelog" label.

[bjsplat]

Snapshot stored with reference name: refs/pull/15106/merge

Test environment: https://babylonsnapshots.z22.web.core.windows.net/refs/pull/15106/merge/index.html

To test a playground add it to the URL, for example:

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/15106/merge/index.html#WGZLGJ#4600

Links to test babylon tools with this snapshot:

https://playground.babylonjs.com/?snapshot=refs/pull/15106/merge https://sandbox.babylonjs.com/?snapshot=refs/pull/15106/merge https://gui.babylonjs.com/?snapshot=refs/pull/15106/merge https://nme.babylonjs.com/?snapshot=refs/pull/15106/merge

To test the snapshot in the playground with a playground ID add it after the snapshot query string:

https://playground.babylonjs.com/?snapshot=refs/pull/15106/merge#BCU1XR#0

[bjsplat]

Visualization tests for WebGPU (Experimental) Important - these might fail sporadically. This is an optional test.

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/15106/merge/testResults/webgpuplaywright/index.html

[bjsplat]

WebGL2 visualization test reporter:

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/15106/merge/testResults/webgl2playwright/index.html

[bjsplat]

Visualization tests for WebGPU (Experimental) Important - these might fail sporadically. This is an optional test.

https://babylonsnapshots.z22.web.core.windows.net/refs/pull/15106/merge/testResults/webgpuplaywright/index.html

[MiiBond]

Would it be possible to get some general feedback about whether this is the right approach for implementing something like this in Babylon? I've modeled this after the DepthPeelingRenderer, which is a scene component, but maybe this should be a pipeline? The effect has many passes and parts so I'm curious whether I'm structuring them in a reasonable way inside Babylon.

[Popov72]

I'll take a look at it early next week.

[Popov72]

Here are some quick comments, but not a full review, as I probably won't be able to follow up because I'll be away for a few months in a couple of days, and it's only in draft mode:

• "clip" in some variable / define names (like viewDepthClipSpace or PREPASS_CLIPSPACE_DEPTH) would probably be better replaced by "ndc"
• I would have created a sub-directory under Rendering/ and move all IBL shadows related files to this directory
• Try to avoid GC in methods called each frame (as the update() method)
• All shader code should probably be in Shaders/ and not directly in .ts files (cf. iblShadowsImportanceSamplingRenderer.ts, which seems to be the only file concerned, probably because you're still working on the implementation)
• Perhaps we should find an abbreviation other than "sss" for "screen space shadows", because it conflicts with "sub-surface scattering".
• I don't think the current code support dynamic update of the scene (adding/removing meshes) after the ibl shadow renderer has been enabled?

As for your question about implementation, I think a rendering pipeline would indeed be better suited, as it would better isolate the code from the rest of the engine. We're going to try and move away from the existing architecture with scene components, so it's best not to rely on them...

I also wonder about performance... With the default values of 256 for the voxel resolution, and assuming 8 draw buffers are supported, it means (256/8)*3=96 rendering to regenerate the voxelized scene. Even with a resolution of 64, it's still 24 passes to generate the voxelized scene. Have you been able to run tests on different scenes of varying complexity and assess the impact on performance?

[MiiBond]

Excellent, feedback, @Popov72! Thank you.

Regarding performance, yes, it's a lot of draw calls to generate the voxel grid and it isn't really suited to dynamic scenes for that reason (at least in it's current form). There are optimizations yet to be made but, to be honest, most of the work I have ahead of me is for improving the quality rather than the performance.

Enjoy your time off!

[MiiBond]

I turned this into a pipeline but I'm not yet using the addEffect functionality and adding the pipeline to the scene.postProcessRenderPipelineManager. I think most of the per-camera stuff that I have can be converted to this system (many are ProceduralTextures right now) but I'm unsure of my AccumulationPass. Right now, it relies on a couple of buffers from the previous frame, including the previous result of the accumulation pass. Is there a way to store off RT's to then be used for the next frame for each camera? It doesn't look like the PostProcessRenderPipeline was designed with this in mind.