Custom projection matrix breaks the shadow

[Th3V1kt0r]

System Information

• Ogre Version: master (10.02.2022)
• Operating System / Platform: Win7
• RenderSystem: OpenGL
• GPU: GeForce GTX 1060 6GB

Detailled description

If I use my own projection matrix, then the shadow is displayed incorrectly.

You can also see the error if you take Sample_ShadowMapDebugging and use Ogre's own projection matrix as custom projection matrix.

For example like this:

diff --git a/Samples/2.0/ApiUsage/ShadowMapDebugging/ShadowMapDebuggingGameState.cpp b/Samples/2.0/ApiUsage/ShadowMapDebugging/ShadowMapDebuggingGameState.cpp
index 52db386537..b5d59f95ae 100644
--- a/Samples/2.0/ApiUsage/ShadowMapDebugging/ShadowMapDebuggingGameState.cpp
+++ b/Samples/2.0/ApiUsage/ShadowMapDebugging/ShadowMapDebuggingGameState.cpp
@@ -170,6 +170,10 @@ namespace Demo
                                                gaussianDeviationFactor, K );
 #endif

+        Ogre::Camera* camera = mGraphicsSystem->getCamera();
+        Ogre::Matrix4 proj = camera->getProjectionMatrix();
+        camera->setCustomProjectionMatrix(true, proj);
+
         TutorialGameState::createScene01();
     }
     //-----------------------------------------------------------------------------------

[Th3V1kt0r]

It seems that the shadow camera does not work properly when the projection matrix is custom. If I define the projection matrix as not custom, then the shadow looks reasonable. Example of how it works for me:

diff --git a/OgreMain/src/OgreShadowCameraSetupPSSM.cpp b/OgreMain/src/OgreShadowCameraSetupPSSM.cpp
index 9cc717440d..3db59b1333 100644
--- a/OgreMain/src/OgreShadowCameraSetupPSSM.cpp
+++ b/OgreMain/src/OgreShadowCameraSetupPSSM.cpp
@@ -150,6 +150,13 @@ namespace Ogre
         _cam->setFarClipDistance( farDist );
         _cam->setCullingFrustum( NULL );

+        const Matrix4 oldProjection = cam->getProjectionMatrix();
+        const bool iscustom = cam->isCustomProjectionMatrixEnabled();
+        if (iscustom)
+        {
+            _cam->setCustomProjectionMatrix(false, oldProjection);
+        }
+
         if( iteration < mNumStableSplits )
         {
             mConcentricShadowCamera.getShadowCamera( sm, cam, light, texCam, iteration,
@@ -163,6 +170,11 @@ namespace Ogre
                                                        viewportRealSize );
         }

+        if (iscustom)
+        {
+            _cam->setCustomProjectionMatrix(true, oldProjection);
+        }
+
         // restore near/far
         _cam->setNearClipDistance( oldNear );
         _cam->setFarClipDistance( oldFar );