[MeshStandardMaterial] envMap reflections are not occluded by aoMap

[Silvaire]

Hello,

In the following scenes rendered using the MeshStandardMaterial and a white ambientLight, you can see

• an object using an aoMap without any envMap (so no dynamic reflexion):
• an object using an aoMap with an envMap (so with dynamic reflexion):

The problem is that in the second case (which is the one I'm trying to achieve), the light from the envMap (or reflexions) is not occluded by the aoMap. This leaves the rendering with bright regions that should be dark (for example, behind the cushion on the right side, there's light between the cushion and the back of the seat. According to the aoMap, it should be completely dark - as it is when I turn of the envMap like in the first case).

I saw a few discussions on the topic, specifically in https://github.com/mrdoob/three.js/issues/7377, but I was wondering if

• I understood right, and what I'm seeing is currently expected for three.js?
• this behavior is going to be improved/fixed in the future?

Thanks in advance for your help,

Silvain Toromanoff

[Silvaire]

Even more specifically, this comment in https://github.com/mrdoob/three.js/issues/7377 from WestLangley seems to sum up my current issue:

envMap, which is a form of an image-based light that reflects specularly, should be modeled as indirect specular. We will get there eventually. When we do, we will decide if specular light reflected from an indirect light source should be occluded by the aoMap, too -- or by how much.

Has anything new been decided or implemented since then (19.10.15)?

[mrdoob]

/ping @WestLangley

[WestLangley]

In r.74, the aoMap (ambient occlusion map) occludes diffusely-reflected light from ambient (i.e., indirect) light sources.

In MeshStandardMaterial, the sources of diffusely-reflected indirect light include AmbientLight, LightMap, HemisphereLight, and the diffuse component of light from the envMap. (See note below).

This means that specularly-reflected light from the envMap is not occluded by the aoMap. That could be changed, but I would want to have a compelling reason to do so -- such as, evidence that is how it is done in other libraries.

Even if we were to make that change, specularly-reflected light from direct light sources, such as spot lights, would still not be occluded by the aoMap because the aoMap only occludes ambient (indirect) light.

/ping @bhouston

---

Note: Because the environment cube does not blur across seams of the cube map, the diffuse component of the envMap does not look very good, and in r.74, that term is commented out of the shader. Once we implement proper blurring, the diffusely-reflected component of light from the environment map will be added back into the model.

[bhouston]

@WestLangley wrote:

Because the environment cube does not blur across seams of the cube map, the diffuse component of the envMap

Remember that @spidersharma03 made a pull request that addresses that.

Regarding specular occlusion it is discussed in the Frostbite paper on pages 75, 76 with an image on page 77 here:

http://www.frostbite.com/wp-content/uploads/2014/11/course_notes_moving_frostbite_to_pbr.pdf

[Silvaire]

Thanks for your answers and help!

I have rather little knowledge of 3D rendering, as I am pretty new to the game, so the following might be missing the point. The fact that the seemingly common situation I am trying to achieve needs the light from the envMap to be occluded by the aoMap to achieve a realistic result seems to be a "compelling reason" to me. Unless there's another way to get dynamically reflected lights only in the proper regions.

From a more technical point of view, and if I'm not mistaken, the link shared by @bhouston shows that Frostbite is trying to occlude the specularly-reflected lights from reflexions, because they leave "artifact" just like the one visible in my screenshots above. It seems relatively more complex to achieve acceptable results than with diffusely-reflected indirect light, but nevertheless something that would improve the accuracy of the final rendering.

[Silvaire]

Out of curiosity, I recreated the same scene (white ambientLight, no other lights) in Unity to check the behaviour when playing with the settings.

In this first screenshot, the occlusion map is ON

In this second one, it is OFF

The smoothness being at 0.5, the reflected lights from the HDR image used as a reflection source (~aka envMap in three.js) is both diffuse and specular. As you can see, the occlusion map occludes both diffuse and specular lights from the reflexion (as occluded regions are still completely dark with the occlusion map activated), which tends to prove that Unity also occludes specularly-reflected lights with the occlusion maps.

---

Side note: Even when turning the smoothness to 1 to get almost all reflected light as specular (I'm not 100% sure this is correct, but that is how I understand the setting. Please correct me if I'm wrong), the occlusion map still occludes all light in the appropriate regions, backing my claim even further.

[Silvaire]

Just being as thorough as possible, I now went on to check Sketchfab's editor as well :)

They have a toggle for occluding specular (indirect lights only, I assume) with the occlusion map

Here is the result for my couch, with occlude specular the toggle ON:

And with the occlude specular toggle OFF:

This is also behaving as expected, showing that this option - if not necessarily a standard in other libraries - is at least possible.

[WestLangley]

Thank you @bhouston and @Silvaire for your replies.

@Silvaire can you share your model/scene by any chance for testing?

I agree this needs to be fixed. Perhaps the Frostbite approach will be acceptable, even though, as they explain in the paper, it is "visually pleasant", and not physically-based.

I should also mention the obvious: in fixing artifacts, we need to be careful to implement a solution that is flexible enough to work in general -- not for just one particular model.

[Silvaire]

I agree with your obvious statement, as it wouldn't make any sense to have a "solution" designed only for my specific model. Especially since I am probably going to encounter a similar situation for about 20 various models, so a general fix - whether it is just physically realistic or not - would be the only viable option.

I can share my model's .obj files (it's in two parts), texture and occlusion map privately to you, if you want to be able to test the behaviour. Can you provide an email or anything that I can send the files to?

[Silvaire]

I just followed you on Twitter now.

I also did a bit more of reading in https://github.com/mrdoob/three.js/pull/6263, and I am just a little curious about the reason of this:

To be specific, the way that ambientOcclusionMap has been implemented is that it modulates the outgoingLight after the main lighting equation, thus it modulates both diffuse and specular and emissive contributions.

This sounds to me like it was working as expected a while back, and somehow, you decided to remove the specular component. Why was that done, and if the reason is valid, why would we put it back now?

However, in https://github.com/mrdoob/three.js/issues/6271, this seems to be a possible compromise for the situation?

Perhaps just exposing some flags like AOAffectDirect, AOAffectAmbient, AOAffectSpecular (reflection) and include all in the shader?

[WestLangley]

This sounds to me like it was working as expected a while back, and somehow, you decided to remove the specular component. Why was that done?

The aoMap was occluding all light sources previously, which was incorrect. Now it occludes only indirect diffuse. How to handle indirect specular remains to be determined.

[Silvaire]

Hello,

It seems that we reached an agreement to fix the issue? Did I understand that correctly? If yes, do you have any idea of a timeframe for the implementation of a fix?

Thanks for your help again!

[WestLangley]

@Silvaire A solution has been implemented. Can this be closed now?

[bhouston]

@WestLangley Do we have any examples with an oaMap? I couldn't find any.

[bhouston]

@WestLangley I'll ask an artist friend of mine for one and I'll add an example if he can help me out.

[bhouston]

@WestLangley My artist friend, Chris Covelli [1], can get us a sculpted head in the next little while with both a normal map and a oaMap. He just has a bunch lying around. :)

[1] http://www.polygonpusherinc.com/index.php/gallery

[WestLangley]

Do we have any examples with an aoMap? I couldn't find any.

@bhouston http://threejs.org/examples/webgl_materials_displacementmap.html uses an aoMap with MeshPhongMaterial.

[MasterJames]

His crow head gets my vote! Super models all around.

[MasterJames]

Displacement map example above goes from funky effect to flat dark brown with any mouse event rotation on Samsung chrome android mobile.

[Silvaire]

Thank you @WestLangley for the fix. I'll try it out and see if this works for me. I'll keep you posted!

I will re-open if the fix doesn't work in a more general context.

[bhouston]

@WestLangley @MasterJames @mrdoob Chris Covelli offered his Rukus goblin head model for inclusion in the ThreeJS project if we want it: https://clara.io/view/9d11969e-eef5-483f-be63-b702a66bd7bf It has an AO and normal map. I think it is generally more interesting model than the current displacement map guy... (but it can not replace the displacement map guy because it doesn't have a displacement map) Your call.

[mrdoob]

I never say no to models! 😀