Changed lighting calculations from being in tangent space to being in…

[Candoran2]

… view space to reduce problems with interpolation.

This caused shading issues to show up on meshes that didn't show them in-game or in the Creation Kit.

Test mesh: DetailCloset.zip TestCubeMap.zip

Additional changes:

• The cube map intersection vector is now properly brought into world space by first bringing it from view space into model space (inverted of matModelView), and then into world space by multiplication with matModel. It is also no longer normalized (not necessary for that use, and it's not used for anything else).
• Calculation of mv_normalMatrix happens globally, rather than in the vertex shader, since it was calculated only from a uniform matrix.
• World space normals are multiplied with mv_normalMatrix instead of matModelView, since they do the same rotation, but mv_normalMatrix does scale adjustment appropriate for normals (only really noticeable with non-uniform scale, not sure how often that happens in OS, but since the matrix is available anyways I thought it best practice).

As a note, I also noticed that normalization of ViewDir and the light directions happens in the vertex shader instead of in the fragment shader. In principle, this should happen in the fragment shader to prevent some warping on interpolation, but that of course only matters if that lightdir is different between different vertices, so I'm not sure how high priority that is. https://forum.unity.com/threads/why-cant-i-normalize-view-direction-in-vertex-shader.636205/

[ousnius]

@Candoran2 Interesting, I saw your changes to the NifSkope shaders as well.

Can you make sure that mv_tbn is properly set to identity for meshes with model-space textures (bModelSpace)? Right now it's not being initialized.

Did you do tests with a cubemap mesh (with a cubemap specifically created to test the reflection) to see if the reflection is still going into the right directions depending on the mesh, lighting and viewing angle?

[Candoran2]

@Candoran2 Interesting, I saw your changes to the NifSkope shaders as well.

Can you make sure that mv_tbn is properly set to identity for meshes with model-space textures (bModelSpace)? Right now it's not being initialized.

Did you do tests with a cubemap mesh (with a cubemap specifically created to test the reflection) to see if the reflection is still going into the right directions depending on the mesh, lighting and viewing angle?

mv_tbn is only used in the transformation of the normal texture from tangent space to view space. This only happens when the texture is in tangent space (or rather, in the else block for bModelSpace), so I didn't see the point in initializing the matrix for model space textures when it's not used then.

As for the environment reflection, I did test it looked mostly okay. Upon closer inspection, it seems there is a change:

However, the change seems to be for the better. Though the orientation isn't exactly the same, at least now there are different parts of the environment map shown when you rotate around the object.

I think the difference between the OS and NifSkope (which seems to have the same orientations as the CK) is that NifSkope brings the reflected vector into world space, but OS multiplies by matModelView (which, if I'm reading it correctly, brings things from model space into view space, but the reflected vector is already in viewspace).

I'll see if I can make a fix for it tomorrow.

[Candoran2]

I think I've got a correctly working cube map now. The directions are still not the same as in the CK/NifSkope, but I think that's due to a different axis system - when looking at the arrows of the transform tool, the colours for the cubemap match up with the corresponding directions of the axes in NifSkope/the CK.

[ousnius]

@Candoran2

As a note, I also noticed that normalization of ViewDir and the light directions happens in the vertex shader instead of in the fragment shader. In principle, this should happen in the fragment shader to prevent some warping on interpolation, but that of course only matters if that lightdir is different between different vertices, so I'm not sure how high priority that is.

Yeah, the four light directions are the same for every vertex, so it doesn't matter for now. In fact it might save some (very) minor performance to only do it in the vertex shader. The view direction however is the negated vertex position without the projection matrix (camera) applied, so it does change from vertex to vertex, shouldn't cause any artifacting though.

Sorry for the late reply, but would you mind doing the same tweaks for the OB and FO4 shaders? Otherwise, I will have a go at it later.

[Candoran2]

@Candoran2

As a note, I also noticed that normalization of ViewDir and the light directions happens in the vertex shader instead of in the fragment shader. In principle, this should happen in the fragment shader to prevent some warping on interpolation, but that of course only matters if that lightdir is different between different vertices, so I'm not sure how high priority that is.

Yeah, the four light directions are the same for every vertex, so it doesn't matter for now. In fact it might same some (very) minor performance to only do it in the vertex shader. The view direction however is the negated vertex position without the projection matrix (camera) applied, so it does change from vertex to vertex, shouldn't cause any artifacting though.

Sorry for the late reply, but would you mind doing the same tweaks for the OB and FO4 shaders? Otherwise, I will have a go at it later.

Sure, I'll start work on that.

[Candoran2]

I think they should be good now. I haven't tested it much, though, since I don't own FO4. The calculation of the blue channel after normalization seems an odd choice to me, though.

[ousnius]

Looks good to me, thanks for contributing.