Issues with TUFX Ambient Occlusion in IVA mode

[JonnyOThan]

The default TUFX AO settings show artifacts in IVA mode with deferred along the edges of the windows/depth masks:

This isn't terribly surprising since the depth mask isn't really any different from opaque geo - can a stencil fix that?

This artifact goes away with ScalableAmbientObscurance, but in fact that mode seems to not work at all unless "Ambient only" is turned off:

intensity 0.5:

intensity 10:

Only some of the emissives are affected (navball, lit buttons).

With ambient only off:

So that last image really shows the problem: the high-frequency block of switches in the lower center of the screen and the overhang above the navball have relatively lower AO intensity than other parts of the cockpit. Normally these are the areas that should have the strongest shadowing.

Well this is interesting....if I play with the "radius" field it gets closer to what I'd expect but it almost seems to control a direction:

[LGhassen]

Is this with the new FreeIVA depthmasks we discussed last time?

Stencil cannot fix this because Unity doesn't actually give control over the background stencil bit (only a few stencil bits are actually exposed to custom shaders).

You could get around this by checking the alpha channel of the emission texture (_CameraGBufferTexture3) which is set to 0 if nothing opaque rendered (excluding depth masks) and set to 1 if something opaque rendered. I do this here for depthMasks in the lighting pass https://github.com/LGhassen/Deferred/commit/ed34eba4297e511bc0b02a23976c94a71f71a73b#diff-9acef8ad85e71784189f9f00996bcad4126522229d84d6d64647b77bcf43aeabR140

You'll have to add this in the AO shaders. I'm not sure if this will work in HDR or not, you could take a renderdoc/nsight capture with HDR on/off and check if that channel is set to the expected values before doing any work.

[JonnyOThan]

Yeah, this is the current public build of FreeIva with the additional depth mask material on the window.

I'm kind of less concerned about the window than the other stuff. It's actually hard to get the switches and things to behave like they should (the last picture is closer). I'm wondering if the "ambient only" flag isn't working properly with deferred or if something else is going weird about how the whole thing works.

[LGhassen]

I'm really not sure what you mean, the buttons not looking right seems more of a config issue for the radius/intensity.

I'm not sure what the "ambient only" flag does, but by default in the older tufx version and using deferred the AO works correctly in that it only writes the AO to the AO channel (alpha channel of _CameraGBufferTexture0) and that's correctly integrated in the lighting and not just used as a darkening filter over the screen. That's something I really like and why I recommended using TUFX AO with deferred from the start.

Not sure what may have changed since then but I'll retest with the new tufx and freeIVA. Now that I think about it, in the above screens you posted it doesn't look like deferred is installed or working, or maybe that's because I never use without scatterer, not sure. Can you use deferred's debug mode and look at the occlusion mode?