Incorrect RH Orthographic Matrix

[rnvannatta]

The right handed orthographic matrices do not work as expected.

They should return depth between 0 and 1 for z values between znear and zfar, like the left handed ones do, but they do not.

The issue traces to DirectXMath/Inc/DirectXMathMatrix.inl commit 7c30ba5, lines 2826 and 3014. They assign the 4th row 3rd column element to fRange * fNear but I believe the correct value is - fRange * fFar .

For explicitness, I believe the full correct term for M_33 is 1 / (near - far) and the full term for M_43 is -far / (near - far)

I confirmed this value by applying the * 0.5 + 0.5 matrix to the RH orthographic matrix taken from opengl superbible 7e.

[walbourn]

The implementation of XMMatrixOrthographicRH has been the same since the original release. The code has been modified in various ways, but functionally it's been unchanged since the original "xboxmath". I'll have to dig into this a bit...

[walbourn]

The implementation of XMMatrixOrthographicRH uses:

float fRange = 1.0f / (NearZ - FarZ);
M.m[2][2] = fRange;
M.m[3][2] = fRange * NearZ;

The implementation of XMMatrixOrthographicLH uses:

float fRange = 1.0f / (FarZ - NearZ);
M.m[2][2] = fRange;
M.m[3][2] = -fRange * NearZ;

Otherwise the values are the same.

[walbourn]

Legacy D3DXMath used for D3DXMatrixOrthoRH:

pOut->_33 = 1.0f / (zn - zf);
pOut->_43 = pOut->_33 * zn;

for D3DXMatrixOrthoLH:

pOut->_33 = 1.0f / (zf - zn);
pOut->_43 = -pOut->_33 * zn;