Limiting the amount of spot light on scene

[VeniGogniti]

SHORT DESCRIPTION:

In scene I have more than 4 spot light but when I press play only 4 of them are visible, others not. After switching off one of the lights, the next ones are switched on during play. NearDay.zip

Armory: 06 win64 Operating system: win 7 Graphics card & driver: Radeon RX580

[madfatihid]

AFAIK it was intended to be maxed at 4. You can change the max size manually by changing the size of maxLights and maxLightsCluster in LightObject.hx, but it will slow down the rendering as the looping mechanism of lights is still need to be fixed.

[N8n5h]

Any ideas where to start looking to solve this one and fix the problem with lights, @luboslenco ?

[luboslenco]

IIRC the main pain is handling the shadow maps and all the light types. The cap is set to 16 lights total and 4 lights per cluster. The view is divided into 16x16x16 clusters.

Haxe part: https://github.com/armory3d/iron/blob/master/Sources/iron/object/LightObject.hx#L30

Shader part: https://github.com/armory3d/armory/blob/master/Shaders/std/clusters.glsl#L2

A good start could be to simplify uniforms passing: https://github.com/armory3d/armory/blob/master/Shaders/std/light.glsl#L33

// Instead of
uniform mat4 LWVPSpot0;
uniform mat4 LWVPSpot1;
uniform mat4 LWVPSpot2;
uniform mat4 LWVPSpot3;

// Do
uniform mat4 LWVPSpot[maxLightsCluster];

Passing array of mat4 uniforms should nowadays work in Kha.

// Instead of
if (index == 0) {
    vec4 lPos = LWVPSpot0 * vec4(p + n * bias * 10, 1.0);
    direct *= shadowTest(shadowMapSpot[0], lPos.xyz / lPos.w, bias);
}
else if (index == 1) {
    vec4 lPos = LWVPSpot1 * vec4(p + n * bias * 10, 1.0);
    direct *= shadowTest(shadowMapSpot[1], lPos.xyz / lPos.w, bias);
}

// Do?
vec4 lPos = LWVPSpot[index] * vec4(p + n * bias * 10, 1.0);
direct *= shadowTest(shadowMapSpot[index], lPos.xyz / lPos.w, bias);

The problem is the above will cause trouble with WebGL. Alternative solution is to bake shadow maps into single atlas.

Here is where light matrices are bound: https://github.com/armory3d/iron/blob/master/Sources/iron/object/Uniforms.hx#L785

Here is where shadow maps are bound: https://github.com/armory3d/armory/blob/master/Sources/armory/renderpath/Inc.hx#L42

If that gets resolved we can investigate raising maxLights and maxLightsCluster.

[N8n5h]

I gave it a go and tried to simplify the uniform passing and here is my progress so far, I think the uniform part works in Krom at least, but I think I have a problem with index, i.e., if I change this part from light.glsl:

if (index == 0) direct *= PCFCube(shadowMapPoint[0], ld, -l, bias, lightProj, n);
else if (index == 1) direct *= PCFCube(shadowMapPoint[1], ld, -l, bias, lightProj, n);
else if (index == 2) direct *= PCFCube(shadowMapPoint[2], ld, -l, bias, lightProj, n);
else if (index == 3) direct *= PCFCube(shadowMapPoint[3], ld, -l, bias, lightProj, n);

to simply

direct *= PCFCube(shadowMapPoint[index], ld, -l, bias, lightProj, n);

then shadows stop working in my simple scene with 4 point lights. I don't understand much of how index is calculated, so my question is: can index go outside the array range? I tried clamping it but still it didn't work.

int li = int(texelFetch(clustersData, ivec2(clusterI, i + 1), 0).r * 255);

The problem is the above will cause trouble with WebGL

Can you elaborate on this?

Alternative solution is to bake shadow maps into single atlas.

Have you eyed any implementation/presentation in particular?

@luboslenco

[luboslenco]

can index go outside the array range?

It should never get higher than maxLightsCluster - 1, and maxLightsCluster is defined to 4.

I am not sure the "name": "LWVPSpot[0]", "link": "_biasLightWorldViewProjectionMatrixSpot0" can work? It might need to be done via setFloats instead.

Example on how array of vectors is passed: https://github.com/armory3d/armory/blob/master/Shaders/deferred_light/deferred_light.frag.glsl#L39 https://github.com/armory3d/armory/blob/master/Shaders/deferred_light/deferred_light.json#L32 https://github.com/armory3d/iron/blob/master/Sources/iron/object/Uniforms.hx#L640

The problem is the above will cause trouble with WebGL

Can you elaborate on this?

Dynamic array index was not allowed in WebGL. Maybe this is no longer the case for WebGL2? https://stackoverflow.com/questions/30585265/what-can-i-use-as-an-array-index-in-glsl-in-webgl

Alternative solution is to bake shadow maps into single atlas.

Have you eyed any implementation/presentation in particular?

I recall reading about it in the presentation below (in the lighting section), unfortunately it's not very detailed. http://advances.realtimerendering.com/s2016/Siggraph2016_idTech6.pptx http://advances.realtimerendering.com/s2016/Siggraph2016_idTech6.pdf

[N8n5h]

can index go outside the array range?

It should never get higher than maxLightsCluster - 1, and maxLightsCluster is defined to 4.

Ok, maybe it has something to do with this https://stackoverflow.com/a/60110986

I am not sure the "name": "LWVPSpot[0]", "link": "_biasLightWorldViewProjectionMatrixSpot0" can work? It might need to be done via setFloats instead.

According to this and some of my tests it seems to be working, at least on Krom. I've avoided setFloats because of having to write that array, I thought that maybe with the offset I could get past the extra loop when passing the values.

The problem is the above will cause trouble with WebGL

Can you elaborate on this?

Dynamic array index was not allowed in WebGL. Maybe this is no longer the case for WebGL2? https://stackoverflow.com/questions/30585265/what-can-i-use-as-an-array-index-in-glsl-in-webgl

Ok, if the if/else if branching doesn't cause much performance difference than dynamic indexing, then it's possible to go in that route with the uniform setting with offset, right? What do you think?

I will also still be looking towards implementing shadowmap atlases, but it will take me some time since I'm still figuring out how that part of the rendering works in the engine.

[N8n5h]

Update: I got some free time again so I decided to continue with this.

I'm currently trying move the spot lights wvp matrices to a more "dynamic" solution so it's no longer constrained to just 4,

I am not sure the "name": "LWVPSpot[0]", "link": "_biasLightWorldViewProjectionMatrixSpot0" can work? It might need to be done via setFloats instead.

so I gave this a go to try to do it with a setFloats approach, and this is my progress so far https://github.com/N8n5h/armory/tree/light-fix-setfloats, https://github.com/N8n5h/iron/tree/light-fix-setfloats I'm currently facing an issue that I'm not very sure why it's not working, but shadows stop working using the same logic but instead of an array of matrices bound via "LWVPSpot[0]" using a setfloats approach is not working. I managed to get renderDoc to work and I could check that the uniform is passed correctly and all the values are the same in both cases, so unless there is some typo that I didn't manage to catch, I'm not sure what's going on. Do you have any idea if setfloats uniforms are treated differently

This is my test file, I included renderDoc captures for both cases. lights_tests.zip

If no solution rises up I will alternatively try to advance my original approach to make it more dynamic. The problem I face is that I'm not sure where the "dynamism" should be. Should the deferred_light.json file be parsed in python and then write dynamically a version with all the uniforms, or should it be done in iron... what do you think?

Also, another thing I was wondering... is it possible to make Zui draw the shadow map depth texture in gray scale or other than just plain red? It would help a lot with debugging to see what is going on in the texture :sweat_smile: ...

@luboslenco

[N8n5h]

Progress update: I managed to get RenderDoc to confirm that the atlasing solution for shadow maps I wrote is working

So now I'm trying to find a way to utilize this atlas data in the deferred shader...

I'm "following" along this tutorial which helped me writing some parts of the solution https://catlikecoding.com/unity/tutorials/scriptable-render-pipeline/spotlight-shadows/

[N8n5h]

Progress update:

I managed to finally get the deferred/light/shadow shaders to work with an atlas instead of an array of shadowmaps for spot lights...

Now that I got that to work I will move to implement support for some of the trickier ones, directional lights (sun) which already has some sort of atlasing for cascading so I have to take that into account, and point lights, which have a lot of images per light so it may need to be approached differently https://www.gamedev.net/forums/topic/684019-point-light-shadows-in-shadow-atlas/

From there I still need to work

• in the current limitation of 4 light matrices to be sent to the deferred shader
• optimize the algorithm that I wrote for managing atlases which currently uses recursion in a lot of places.
• cover corner cases such as, what happens to the atlas if you remove lights?
• investigate how to make it work for forward

and then finally investigate the "in-fighting" between different lights shaders.

[N8n5h]

Progress Update:

I finally found out what was wrong with my float array approach for spot lights matrices and managed to unlock the limit for those with ease. So at last 16 spot lights using an atlas of 4096x4096:

I'm currently working into adding support for point lights, which can be tricky since it may require a move from cubemaps to a simple texture so it can be part of the atlas. https://www.gamedev.net/forums/topic/687535-implementing-a-cube-map-lookup-function/

[N8n5h]

Progress Update:

I managed to add support for point lights in the atlas solution, by faking a cubemap lookup, so this is the result, 9 point lights in the same texture: There is still some things that I need to solve because it's has a few artifacts, mostly related to bias I think. The atlas is 8192x8192 without dynamic tile sizes, which could be very big for some targets, so I'm trying to add support for dual paraboloid shadow mapping as an alternative for those that really need to save space but want a considerable numbers of lights showing up with acceptable performance and don't mind the quality loss and probably some artifacts.

While I was testing around I also noticed an issue with the rendering, which from my tests I think it's related to the clustering algorithm, since I could reproduce it in "vanilla" armory. The issue seems to appear when you are far enough with more than 2 point lights rendering (that's what I tested so far), the more the lights, the more noticeable the issue gets.

I will try to learn and understand how the clustering algorithm work to try to get this solved, but @luboslenco is this a known issue with the algorithm, or there is something else that you suspect could be causing it?

[luboslenco]

Super cool!

The issue seems to appear when you are far enough with more than 2 point lights rendering (that's what I tested so far), the more the lights, the more noticeable the issue gets.

Not aware of that but likely issue with clustering like you mention - from what I gather it changes with camera movement?

[N8n5h]

Super cool!

Thanks! :)

Not aware of that but likely issue with clustering like you mention - from what I gather it changes with camera movement?

Yes, camera movement and distance. I noticed that the more lights are rendered the closer it appears and the more noticeable it gets

https://streamable.com/y17l2b

[N8n5h]

Progress update:

I'm still trying to get this issue with clustering solved. In the mean time, I extended the support for the atlas for directional lights,

Original left | New right

Unfortunately I ran into some issue that makes the result of cascaded shadows a bit different compared to the original (it can be seen by clicking the image and zooming in). I'm not sure why bundling the shadow maps in a different way could cause such issue but I need to investigate it.

Besides from that, the difference now is that it's trivial to have directional lights share an image with other type of lights.

Edit:

After a lot of testing I found the culprit to the artifact with clusters, and it's this check https://github.com/armory3d/armory/blob/3e819d7e99f2ed9cf4cc751eda4754cd371550a7/Shaders/deferred_light/deferred_light.frag.glsl#L406, it seems in some clusters it's detecting a spot light when there is none, so this produced the artifact. Changing the detection code to something else solved the issue in my tests.

[N8n5h]

progress update:

It took longer than I expected, but after a lot of arduous debugging sessions I finally got LOD to work, or at least to not fail immediately within simple tests. It's still experimental though and probably needs more testing to make sure it's usable.

This is a comparison to show how much space is saved by just varying the subdivisions number in the same scene:

---

The advantage of saving space it's relative though, since the atlas will grow but never shrink (a thing that I'm still pondering if it needs solving or not), so a situation like this can happen if you are exposed to the worst case scenario that is all lights require the max size,

I'm not familiar to what is best to optimize gpu memory in this case, @luboslenco what do you think, should I try to solve this situation or leave it as is?

Explanation of how the LOD works

To save a lot of performance I manually keep count of occupied children tiles for every tile. This allows me to simply use this to discard a tile immediately when looking for free tiles for LOD. Otherwise it would require some complicated algorithm or having to iterate over all the sub-tiles to find out if it can be occupied or not. The size of the shadowmap is determined by taking advantage of the clustering data, similarly to the presentation/paper for clustering, but instead of using the angle I use the minimal Z cluster paired with a custom made function that plots from 1-16, to 1.0-0.0, and basically the closer you get the higher the `shadowMapScale` becomes. So far results I think are acceptable enough and it's a lot faster compared with the solution proposed in the paper. I tried to implement it the way it's done in the paper, but unfortunately I got stuck at the step of obtaining the "solid angle". I couldn't find a way to do it without overcomplicating the clustering algorithm with the data there is available.

I forgot to mention that I also added culling of lights by using the clustering data too.

---

I'm also currently trying to wrap up other sub-projects before opening a PR. Currently porting changes to make the other render paths work, like forward or deferred mobile. Also still trying to get dual paraboloid maps to work.

---

Oh and also, I solved the issue with directional lights, it seemed to be the uniformSize parameter that was incorrect and that lead to that rendering issue. There is still some differences with filtering that I need to investigate, but the result looks more closer now to the original.

[MoritzBrueckner]

@N8n5h I guess this can be closed now?

[N8n5h]

@ N8n5h I guess this can be closed now?

Yes, it can be closed now :+1: