Decompose Matrix Function

[Philip-Trettner]

into Translation, Scale, Rotation, Shear?

See https://glm.g-truc.net/0.9.6/api/a00204.html

[ChargeProduction]

Here is a working example which I have adapted to your library:

        // MIT License - Copyright (C) The Mono.Xna Team
        // This file is subject to the terms and conditions defined in
        // file 'LICENSE.txt', which is part of this source code package.

        /// <summary>
        /// Decomposes this matrix to worldTranslation, worldRotation and worldScale elements. Returns <c>true</c> if matrix can be decomposed; <c>false</c> otherwise.
        /// </summary>
        /// <param name="worldScale">Scale vector as an output parameter.</param>
        /// <param name="worldRotation">Rotation quaternion as an output parameter.</param>
        /// <param name="worldTranslation">Translation vector as an output parameter.</param>
        /// <returns><c>true</c> if matrix can be decomposed; <c>false</c> otherwise.</returns>
        public static bool Decompose(this mat4 mat, out vec3 scale, out quat rotation, out vec3 translation)
        {
            translation.x = mat.m30;
            translation.y = mat.m31;
            translation.z = mat.m32;

            float xs = (Math.Sign(mat.m00 * mat.m01 * mat.m02 * mat.m03) < 0) ? -1 : 1;
            float ys = (Math.Sign(mat.m10 * mat.m11 * mat.m12 * mat.m13) < 0) ? -1 : 1;
            float zs = (Math.Sign(mat.m20 * mat.m21 * mat.m22 * mat.m23) < 0) ? -1 : 1;

            scale.x = xs * (float)Math.Sqrt(mat.m00 * mat.m00 + mat.m01 * mat.m01 + mat.m02 * mat.m02);
            scale.y = ys * (float)Math.Sqrt(mat.m10 * mat.m10 + mat.m11 * mat.m11 + mat.m12 * mat.m12);
            scale.z = zs * (float)Math.Sqrt(mat.m20 * mat.m20 + mat.m21 * mat.m21 + mat.m22 * mat.m22);

            if (scale.x == 0.0 || scale.y == 0.0 || scale.z == 0.0)
            {
                rotation = quat.Identity;
                return false;
            }

            mat4 m1 = new mat4(mat.m00 / scale.x, mat.m01 / scale.x, mat.m02 / scale.x, 0,
                                   mat.m10 / scale.y, mat.m11 / scale.y, mat.m12 / scale.y, 0,
                                   mat.m20 / scale.z, mat.m21 / scale.z, mat.m22 / scale.z, 0,
                                   0, 0, 0, 1);

            rotation = new quat(m1);
            return true;
        }   

The code can be found here: https://github.com/MonoGame/MonoGame/blob/develop/MonoGame.Framework/Matrix.cs#L2081

I tried to decompose a dmat4 which was multiplied hierarchically with its parents matrix like this: dmat4 transform = parent.Transform * localTransform which worked well.

It would be good to make the out variables ref in an other extension method too. So you can define new variables with out and edit some with ref.