Better understanding of computing skinning matrices

First of all, if this is not related at all to glTF & skinning and is more related to blender then i apologize and this issue can be closed.

I am currently trying to understand how the 'final joint matrix' (i.e. the matrix that each influenced vertex is multiplied by) is computed My current knowledge of how it's calculated: jointMatrix = globalMatrix * inverseBindMatrix where globalMatrix is the joint's local matrix multiplied by it's parent global matrix and inverseBindMatrix = invert(globalMatrix)

Now consider a simple setup with just 2 joints: Joint 1: Local matrix = Identity matrix (as i want this joint to have a location of (0, 0, 0), no rotation and a scale of (1, 1, 1) Joint 2: Local matrix = Translation: (100, 50, 50), no rotation, scale(1, 1, 1) of course stored in a matrix but i just wrote it out this way for clarity.

Joint 1 has no parent as it's the root joint, Joint 2 has Joint 1 as it's parent

Now as i understand when a joint in rotated it uses it's local translation as the origin of rotation instead of (0, 0, 0) and here's where my confusion comes from because the math doesn't seem to add up

Say Joint 2 was rotated by 45 degrees in the x axis, what i'd expect the final joint matrix to do when influenced vertices are multiplied by it is rotate each vertex 45 degrees in the x axis around the bone's local translation (i.e. 100, 50, 50 in this case) that seems to be happening when i rotate the bone in blender but the resulting matrix when i do these computations doesn't seem to take that into account and instead assumes rotation around (0, 0, 0)

Here's sample code i wrote that would compute the final joint matrix for the above example:

import lombok.Getter;
import lombok.Setter;
import org.joml.Matrix4f;

public class JointTest {
    public static void main(String[] args) {
        Joint joint1 = new Joint();
        // identity matrix as root should be at position (0, 0, 0), have no rotation or scale
        joint1.setLocalMatrix(
                new Matrix4f()
        );
        Joint joint2 = new Joint();
        // joint 2 should have a position of (100, 50, 50) which should be it's rotation origin
        joint2.setLocalMatrix(
                new Matrix4f().translate(100, 50, 50)
        );
        joint2.setParent(joint1);
        joint2.computeAndSetIBM();

        Matrix4f animationMatrix = joint2.getLocalMatrix()
                .rotateX((float) Math.PI / 4f, new Matrix4f());
        joint2.setLocalMatrix(animationMatrix);

        Matrix4f globalMatrix = joint2.computeGlobalMatrix();
        Matrix4f inverseBindMatrix = joint2.getInverseBindMatrix();

        Matrix4f finalJointMatrix = globalMatrix.mul(inverseBindMatrix, new Matrix4f());
        System.out.println(finalJointMatrix);
    }

    @Getter
    @Setter
    private static class Joint {
        // Matrix4f is from JOML (column-major)
        private Matrix4f localMatrix;
        private Matrix4f inverseBindMatrix;
        private Joint parent;

        public void computeAndSetIBM() {
            Matrix4f global = computeGlobalMatrix();
            inverseBindMatrix = global.invert(new Matrix4f());
        }

        public Matrix4f computeGlobalMatrix() {
            Matrix4f globalMatrix = new Matrix4f(localMatrix);
            if (parent != null) {
                return globalMatrix.mul(parent.computeGlobalMatrix(), new Matrix4f());
            }
            return globalMatrix;
        }
    }
}

The result is:

 1.000E+0  0.000E+0  0.000E+0  0.000E+0
 0.000E+0  7.071E-1 -7.071E-1  5.000E+1
 0.000E+0  7.071E-1  7.071E-1 -2.071E+1
 0.000E+0  0.000E+0  0.000E+0  1.000E+0

Which if i understand correctly is not really a 45 degree rotation in the x axis around the origin (100, 50, 50), at least it does not seem correct but please do let me know if im wrong.

I'll have to allocate a bit more time to understand the code, the maths, and ... what the "core" of the question is.

For now, two pointers:

The question seems to be related to the order of operations and the effects of translations. This also confused me a bit, and there's some discussion and (hopefully) helpful examples about that in https://github.com/KhronosGroup/glTF-Tutorials/pull/64#issuecomment-969058924
The actual skinning computation in JglTF happens around https://github.com/javagl/JglTF/blob/a016307611a5d6360969e54f5379badee46d03da/jgltf-viewer/src/main/java/de/javagl/jgltf/viewer/UniformGetterFactory.java#L441 - it may not be perfectly intuitive and could probably written in a simpler or more intuitive form for a specific example, but might be worth a look

I'll have to allocate a bit more time to understand the code, the maths, and ... what the "core" of the question is.

For now, two pointers:

The question seems to be related to the order of operations and the effects of translations. This also confused me a bit, and there's some discussion and (hopefully) helpful examples about that in Updated skinning example and text KhronosGroup/glTF-Tutorials#64 (comment)

The actual skinning computation in JglTF happens around https://github.com/javagl/JglTF/blob/a016307611a5d6360969e54f5379badee46d03da/jgltf-viewer/src/main/java/de/javagl/jgltf/viewer/UniformGetterFactory.java#L441

it may not be perfectly intuitive and could probably written in a simpler or more intuitive form for a specific example, but might be worth a look

Thank you for the quick reply.

The question seems to be related to the order of operations and the effects of translations.

I believe that is the case indeed.

I have read both the issue you linked about skinning (I did kinda skim over it so i'll likely have to read it again altho it did make a few things more clear) and the code you linked in jgltf viewer which did confuse me a bit, specifically this part:

// Create the joint matrix suppliers. Each of them will provide a
// matrix that is computed as
// [jointMatrix(j)] = 
//     [globalTransformOfNode^-1] *
//     [globalTransformOfJointNode] *
//     [inverseBindMatrix(j)] *
//     [bindShapeMatrix]

Because in the glTF-Tutorials (Skins page): https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_020_Skins.md It mentions this: so the pseudocode for computing the joint matrix of joint j may look as follows:

jointMatrix(j) =
  globalTransformOfJointNode(j) *
  inverseBindMatrixForJoint(j);

1) What is 'globalTransformOfNode' even referring to(or well what is 'Node' in this scenario)? i thought Joints were nodes and that was all there is to it, and why is it inverted. 2) What is a bindShapeMatrix in glTF, i thought that didn't even exist and the structure for joints was the following: Each Joint was a Node which has a local transform (either specified by a local transform matrix or T/R/S components) and an inverse bind transform which should just be invert(globalTransformOfJointNode)

I guess the core of this question is just trying to figure out exactly how the math mentioned in the glTF Tutorials Skin page adds up because i believe i tried to replicate it in the code that i included above altho it does seem wrong (either i just wrote it incorrectly or i misunderstood the Skin page in the glTF tutorials repo)

My goal right now(which im struggling with) is to create the simplest possible example (a root joint at 0, 0, 0 and another joint(Joint 1, whose parent is the root joint) somewhere else) then a simple 1 keyframe animation that just rotates joint 1 in blender. the model itself would also just be a triangle(so 3 vertices) and each one would be influenced by joint 1. Then import that into my program and do the necessary math to construct the matrix that each vertex is multiplied by and then just display it.

That is just to understand the math behind skeletal animation/skinning tho, my actual goal is to take an animated model in blender, export it to glTF then import it into my program and instead of doing the normal procedure which computes the joint matrix that influenced vertices are multiplied by, i want to actually decompose it into a set of transformations (translation, rotation, scale) and origin (for the origin of rotation/scale) an example of what i mean: Imagine the same setup as above (Root joint and Joint 1) the local matrix of the root joint would be the identity matrix and joint 1 would have a translation of (100, 50, 50) no rotation or scale (i.e. scale of 1, 1, 1) Now if i made a one keyframe animation that just rotates Joint 1 by 45 degrees in the x axis then the decomposed transformations for keyframe 1 that i'd want are the following: Transformation 1: Set origin of rotation to (100, 50, 50) Transformation 2: Apply a 45 degree rotation in the x axis, the engine code would do this when applying the animation:

setOrigin(100, 50, 50)
rotateX(45)

void rotateX(int angle) {
    translate(-origin) // translate to origin for rotation
    rotateX(angle)
    translate(origin) // put back in position
}

Why? the engine i am using is quite old and that's how they do animations (why am i using such an old engine? modding purposes, i enjoyed playing the game as a child and now im creating a modded version of it and are currently in the process of writing tools for it)

Regarding the link to the code: I hesitated a bit to post this, because I thought that it might be confusing. That's why I mentioned

it may not be perfectly intuitive and could probably written in a simpler or more intuitive form for a specific example

And your question now suggests that the code may indeed have been confusing:

What is a bindShapeMatrix in glTF

The JglTF library was originally started for glTF 1.0. And when glTF 2.0 came out, I tried to support both glTF 1.0 and glTF 2.0 at the same time. In hindsight, this may not have been the smartest decision. Large parts of the code could be much, MUCH simpler if it only targeted glTF 2.0.

tl;dr: The bindShapeMatrix is only relevant for glTF 1.0. For glTF 2.0, it is no longer used...

I have read both the issue you linked about skinning (I did kinda skim over it

Yes, it was a long dicussion, because I also struggled with some parts, and how to make them as clear as possible in the tutorial example. And it's hard to answer specific questions about the math, or even one specific implementation approach that someone came up with, without having the details "freshly" in memory. (Or to put it that way: I can not look at your code and immediately tell you what is wrong there. I might have had a better chance of doing this one year ago, when this linked issue was actively discussed. But even then it's not always easy)

However, you might have seen that there is some optional "logging" in the code, along the lines of this log option. This means that I can (usually, with reasonable effort) print all these matrices.

For the SimpleSkin model, this prints something like

2022-10-01 15:38:53.325 INFORMATION   : inverseBindMatrix 0:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.325 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.325 INFORMATION   : jointMatrix 0:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.325 INFORMATION   : inverseBindMatrix 1:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, -1.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.325 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.325 INFORMATION   : jointMatrix 1:
-1.1920929E-7, -1.0000001, 0.0, 1.0000001, 
1.0000001, -1.1920929E-7, 0.0, 1.0000001, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : inverseBindMatrix 0:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : jointMatrix 0:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : inverseBindMatrix 1:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, -1.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-01 15:38:53.340 INFORMATION   : jointMatrix 1:
5.9604645E-8, -0.99999994, 0.0, 0.99999994, 
0.99999994, 5.9604645E-8, 0.0, 0.99999994, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0,

for this state:

glTF Skin State

Now... is that helpful? I don't know for sure. But if you have a specific example (as a glTF file), then I might be able to print something like this, for a certain state, so that you could compare it to the result of your computations...

I do fully understand that it's hard to answer a specific question about the math and that it's difficult to just look at the code i posted above and be able to tell me what's wrong. The debug information you posted above would indeed be helpful if you could print it for the glTF file below (it contains one triangle, one bone that affects the triangle and the animation is 3 keyframes, the bone has a translation of (2, 2, 0) so it should rotate around that, which it does visually) If you can also show a picture of how it looks like at keyframe 1 then that'd help too but honestly just the debug info would be more than enough. Key frame 0: default pose (no transformation applied) Key frame 1: rotation applied Key frame 2: default pose (no transformation applied) animatedtriangle.zip

OK. I have added the output for the required time stamps. I did that locally with a very quick and crude hack (there are some issues in JglTF that I never had the chance to tackle), but maybe it's helpful:

Update for time 0.0
2022-10-02 14:13:08.672 INFORMATION   : inverseBindMatrix 0:
1.0, -0.0, 0.0, -2.0, 
-0.0, 1.0, -0.0, 0.0, 
0.0, -0.0, 1.0, 2.0, 
-0.0, 0.0, -0.0, 1.0, 

2022-10-02 14:13:08.673 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-02 14:13:08.673 INFORMATION   : jointMatrix 0:
1.0, 0.0, 0.0, -2.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 2.0, 
0.0, 0.0, 0.0, 1.0, 

Update for time 0.04167
2022-10-02 14:13:08.701 INFORMATION   : inverseBindMatrix 0:
1.0, -0.0, 0.0, -2.0, 
-0.0, 1.0, -0.0, 0.0, 
0.0, -0.0, 1.0, 2.0, 
-0.0, 0.0, -0.0, 1.0, 

2022-10-02 14:13:08.701 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-02 14:13:08.701 INFORMATION   : jointMatrix 0:
1.0, 0.0, 0.0, -2.0, 
0.0, 1.2528896E-4, -1.0, -2.0, 
0.0, 1.0, 1.2528896E-4, 2.5057793E-4, 
0.0, 0.0, 0.0, 1.0, 

Update for time 0.08333
2022-10-02 14:13:08.709 INFORMATION   : inverseBindMatrix 0:
1.0, -0.0, 0.0, -2.0, 
-0.0, 1.0, -0.0, 0.0, 
0.0, -0.0, 1.0, 2.0, 
-0.0, 0.0, -0.0, 1.0, 

2022-10-02 14:13:08.709 INFORMATION   : bindShapeMatrix:
1.0, 0.0, 0.0, 0.0, 
0.0, 1.0, 0.0, 0.0, 
0.0, 0.0, 1.0, 0.0, 
0.0, 0.0, 0.0, 1.0, 

2022-10-02 14:13:08.710 INFORMATION   : jointMatrix 0:
1.0, 0.0, 0.0, -2.0, 
0.0, 1.0, -1.2583438E-4, -2.5166877E-4, 
0.0, 1.2583438E-4, 1.0, 2.0, 
0.0, 0.0, 0.0, 1.0,

(As I said: The bindShapeMatrix is always the identity for glTF 2.0)

Thanks, that helps a lot. Just to confirm the joint matrix logged in ur reply was produced with this right?

Supplier<float[]> jointMatrixSupplier = MatrixOps
.create4x4(nodeModel.createGlobalTransformSupplier())
.invert4x4()
.multiply4x4(jointNodeModel.createGlobalTransformSupplier())
.multiply4x4(inverseBindMatrixSupplier)
.multiply4x4(bindShapeMatrixSupplier)
.log("jointMatrix "+j, Level.FINE)
.build();

Also could you explain what the difference is between these two?

//     [globalTransformOfNode^-1]
//     [globalTransformOfJointNode]

If i had to guess then globalTransformOfJointNode is the current global transform of the joint node and globalTransformOfNode is the global transform of it's initial state, but im not 100% sure. i will likely clone the repository and try to figure it out myself by tomorrow.

The globalTransformOfNode refers to the node that the skin is attached to. This is also something that should always be the identity matrix in glTF 2.0.

Trying to figure out solutions by "reverse engineering" the existing code might not be the best approach, because some aspects of the implementation may be confusing, and there are some 'gotchas' in the part that renders the glTF. For example, the "default material" in the current state does not include skinning functionality, so by default, you won't see the skinned result if there is a default material. For PBR materials, it should work, but the PBR implementation itself is far from perfect.

But of course, you may give it a try and see whether you can gain helpful insights from it.

I agree with you that it's not really the best approach, i'll just take a quick look at it to see if i find anything useful from it. Thank you for all the replies, they actually helped quite a lot. I'll close this issue as i have no other questions currently.

javagl / JglTF

Better understanding of computing skinning matrices #76