[Clarification] Skeleton root and joints hierarchy

[Selmar]

The skins section of the specification mentions: The skeleton property points to node that is the root of a joints hierarchy.

It is unclear to me whether this means that all joints must to part of the hierarchy below the skeleton root node or not, as it says a joints hierarchy and not the joints hierarchy.

I encountered an issue (or, thus, perhaps not?) with the Babylon exporter, which does not always export the skeleton root node as a parent of all the referenced joints.

[najadojo]

The skeleton is rooted (with some additional transforms see #1195 ) with the given node. The joint list are those nodes that participate in the skinning, that is they have inverse bind values and are referred to in the JOINTS_0 mesh attribute by their index in this list. All nodes referenced in the tree described by the skeleton node are part of the skeleton. The nodes without joint index are attachments to the skinned mesh. In the asset you refer to the root is an attachment; BrainStem is a sample that also presents this condition.

[Selmar]

Alright, I understand all of that. In fact, the BrainStem model loads fine for me.

I will pose my problem as a question: Is it allowed for a node in the joints array to NOT be a child (or grand-child, etc.) of the skeleton node?

[bghgary]

Is it allowed for a node in the joints array to NOT be a child (or grand-child, etc.) of the skeleton node?

If #1195 goes in, then the skeleton property should be ignored. We are considering removing the skeleton property if we can converge on the PR.

[donmccurdy]

In addition to #1195, I think the other question here is whether the joints must actually be a strict tree. I.e., even if we remove the skeleton property, it is safe for clients to assume that the joint list is a heirarchy with a single root?

Would this be valid, where Armature is the node (currently) specified by skeleton?

Scene
├── Armature
│   ├── Bone1
│   ├── Bone2
│   └── Bone3
├── Bone4
└── SkinnedMesh

[Selmar]

In the example of @donmccurdy, the skeleton property is indeed useless. In that example, the scene is actually the skeleton root, from a practical point of view. This is what I wanted clarification on, as the exporter I used gave me something like that example. This struck me as an error, but I couldn't find a clear answer.

For some background, in our case, bones and nodes are separate entities. I found it practical to know ahead of time which node I can use as the skeleton root (on which I build the skin), but if I in any case use the scene as a root instead, it would work the same. The only difference, in our case, is that we would have more bones (which is no big deal and can of course be optimized anyway, if required).

[najadojo]

This issue came up again; I modified RiggedSimple.zip to demonstrate for clients that are impacted. It seems Babylon.js and Three.js will both load this scene and find a common root bone for the skin. Our parser is taking the skeleton node to be the root and missing out on the second bone. Cesium also doesn't seem to load this modified asset.

I agree with @Selmar that this is in error do we all agree enough to clarify the spec?

[pjcozzi]

@bghgary any thoughts?

[bghgary]

@pjcozzi Thanks for the ping. Somehow missed responding to this.

load this scene and find a common root bone for the skin

Babylon.js loads all transforms up the tree until it reaches a root node.

we all agree enough to clarify the spec?

+1 for updating the spec. The question is whether to remove the skeleton property or indicate it should be ignored. If removing the property doesn't cause any compatibility issues, that's the one I would choose, but I'm not completely sure it won't cause issues. Can anyone think of a reason why it would be a problem?

[najadojo]

What about the option that skeleton should be the root of a node tree that includes all nodes in the joints list?

If that won't work I suggest we modify the spec to ignore skeleton because there are parsers that require it be present.

[lexaknyazev]

there are parsers that require it be present.

That field is optional in the current schema (and iirc always was optional). How could such parsers exist?

[najadojo]

Because the author didn't read the spec closely enough. :(

[lexaknyazev]

That means that those parsers will fail on valid assets even if we won't change the schema. To fix that we'd need to make that field required and ignored at the same time.

[Raveler]

I've been trying to figure out how exactly I am supposed to implement this, and it's very unclear based on the specs. This is the only other thread where it is expanded upon but it is still totally unclear to me what I am supposed to do, so I'm going to ask a few questions here.

1. If one bone is NOT part of the skeleton node's tree, are we supposed to ignore it or should I include it in the skin anyway? This means I would need to "overrule" the skin.skeleton-value if I discover that it does not contain all the joints in its hierarchy. Is this the expected behaviour?

2. What about this layout:

{
    "scenes": [
        {
            "nodes": [ 0 ]
        }
    ],

    "nodes": [
        {
            "skin": 0,
            "mesh": 0,
            "name": "Scene Root"
        },
        {
            "name": "Bone A"
        },
        {
            "name": "Bone B"
        }
    ],

    "skins": [
        {
            "joints": [ 1, 2 ]
        }
    ],
}

In this case, both joints simply have no shared root in the hierarchy. They are their own root. Is this a valid file? If not, where does it violate the specs? If it is valid, should I calculate the joint matrix for each bone as if it has a different root?

3. Finally, IF I discover that one of the joints is not part of the defined skeleton's hierarchy, there are two things I could do: a) I could consider the skeleton value invalid, and use the node hierarchy root (or multiple roots as in the above example) instead. This I believe is conform the specs (as it says that skeleton defaults to scene root). b) However, as I read in @najadojo 's comment above, Babylon.js and Three.js apparently find a common root in that case, which might not be the scene root.

Example of problem (3) where this could make a real difference:

{
    "scenes": [
        {
            "nodes": [ 0 ]
        }
    ],

    "nodes": [
        {
            "skin": 0,
            "mesh": 0,
            "name": "Scene Root"
        },
        {
            "translation": [2, 0, 0],
            "children": [2],
            "name": "Should this be the skin root? (according to specs?)",
        },
        {
            "children": [3, 4],
            "name": "Or should this be skin root? (three.js and babylon.js behavior)"
        },
        {
            "name": "Bone A"
        },
        {
            "name": "Bone B"
        }
    ],

    "skins": [
        {
            "skeleton": [3],
            "joints": [ 3, 4 ]
        }
    ],
}

[ziriax]

If one bone is NOT part of the skeleton node's tree, are we supposed to ignore it or should I include it in the skin anyway? This means I would need to "overrule" the skin.skeleton-value if I discover that it does not contain all the joints in its hierarchy. Is this the expected behaviour?

As far as I understand (I could be wrong, I'm not an expert; I did write the Maya2glTF exporter, so if I make mistakes here, I hope the experts will correct me, so I can adjust the exporter accordingly)

A joint (aka bone) is just a reference to a node, be it in a hiearchy or not. The only thing that is needed to apply skinning to a mesh is the world (aka global) transform of the joint's node. IMO you don't have to see a skeleton as a hierarchy at all. It's just an array of references to nodes.

What about this layout:

{ "scenes": [ { "nodes": [ 0 ] } ],

"nodes": [
    {
        "skin": 0,
        "mesh": 0,
        "name": "Scene Root"
    },
    {
        "name": "Bone A"
    },
    {
        "name": "Bone B"
    }
],

"skins": [
    {
        "joints": [ 1, 2 ]
    }
],

} In this case, both joints simply have no shared root in the hierarchy. They are their own root. Is this a valid file? If not, where does it violate the specs? If it is valid, should I calculate the joint matrix for each bone as if it has a different root?

It certainly is valid for a skin to have multiple skeleton roots, although the spec then forces you to not define the skeleton property. See attached such a file, it is a cuboid with two root joints, the top joint drags the top part of the mesh, and bottom joint the bottom part.

Finally, IF I discover that one of the joints is not part of the defined skeleton's hierarchy, there are two things I could do: a) I could consider the skeleton value invalid, and use the node hierarchy root (or multiple roots as in the above example) instead. This I believe is conform the specs (as it says that skeleton defaults to scene root). b) However, as I read in @najadojo 's comment above, Babylon.js and Three.js apparently find a common root in that case, which might not be the scene root. Example of problem (3) where this could make a real difference:

Just ignore the hierarchy all together, skinning has nothing to do with a hierachy.

What bothers me is that according to the spec, if the skeleton property is not defined, the scene root is used as the skeleton root. But I have no idea what this skeleton root is supposed to do. The spec says _Client implementations should apply only the transform of the skeleton root node to the skinned mesh while ignoring the transform of the skinned mesh node. In the example below, the position of node_0 and the scale of node_1 are applied while the position of node_3 and rotation of node4 are ignored.

However, the reference guide specifies something different, that guides shows the following formula for a joint-matrix:

jointMatrix[i] = inverse(globalTransform) * globalJointTransform[i] * inverseBindMatrix[i]

This first inverse(globalTransform) just undoes the globalTransform that is already setup as a model-view-matrix, effectively ignoring the transform of the skinned mesh node. But the skeleton root node doesn't come into play in this formula.

To dig a bit deeper, let's write down a possible complete formula of skinning, assuming the globalTransform has already been setup as model-view-matrix.

I'm going to use a different notation, using fractions of geometric coordinates frames.

So

mesh
------
world

is the matrix of the mesh relative to the world, e.g. the globalTransform above.

And

world
------
mesh

is the matrix of the world relative to the mesh, e.g. the inverse(globalTransform).

We can only multiple matrices when the coordinates frames match (from right-to-left in OpenGL, in DirectX and Maya we have to reverse this), e.g.

B      A 
-   *  -
C      B

Is valid because the Bs match, and gives

A 
-
C

With this notation, matrix multiplication can be seen as just changing the coordinates (=matrix) of the frame to a different basis frame. For me this is a handy semantic tool to check if my matrix multiplications make sense. It also allows me to draw the geometric frames on paper, and reason about the formulas. But that's just me :wink:

Let's start with the simple transformation of a vertex position in mesh (aka model) space to world space:

pos     mesh    pos
----- = ----- * ----
world   world   mesh

So the get the position of the vertex pos in world space, we take the local pos in mesh space, and multiply with the mesh matrix in world space.

When skinning comes into play (let's assume a single joint, with weight 100% to start easy), this becomes:

mesh      world   joint     world     mesh     pos
------ * (----- * ------ * (------  * -----) * ----)
world     mesh    world     joint'    world    mesh

  ^        ^        ^              ^            ^
  |        |        |              |          localVertexPos
  |        |        |      inverseBindMatrix
  |        |        |
  |        |     globalJointTransform
  |        |
  |      inverseGlobalTransform
  |
globalTransform
setup as modelViewMatrix

where joint is the current/animating joint coordinate frame, and joint' was the joint coordinate frame when the skin was bound to mesh (before exporting)

Note that the inverse bind matrix simplies to

world     mesh    mesh
------  * ----- = ------
joint'    world   joint'

Semantically the inverse bind matrix holds the coordinates of the mesh as seen through the binding joint. Multiplying a vertex position with the inverse bind matrix gives

mesh     pos    pos
------ * ---- = ------
joint'   mesh   joint'

So it converts this vertex into the local space of the binding joint.

We can simplify the complex formula above as:

mesh      world   joint     world     mesh     pos
------ * (----- * ------ * (------  * -----) * ----)
world     mesh    world     joint'    world    mesh

= (matrix mul is associative, so we can leave out the parentheses)

mesh     world   joint    world     mesh    pos
------ * ----- * ------ * ------  * ----- * ----
world    mesh    world    joint'    world   mesh

= (coordinate spaces cancel out)

                 joint                      pos     
                 ------ * ----------------------
                 world    joint' 

So in the case where joint == joint', we just get the position in world space, as expected. Otherwise the relative change between the current and binding joint is applied, also as expected.

When weights and multiple joints come into play, we get

         N-1
         ---   
mesh     \              world   jnt[j]    world     mesh     pos     pos
------ *  |(weight[j] * ----- * ------ * (------  * -----) * ----) = -----
world    /              mesh    world     jnt[j]'   world    mesh    world
         ---   
         j=0

In more regular notation:

worldPos = globalTransform * sum (weight[j] * inverse(globalTransform) * globalJointTransform[j] * inverseBindMatrix[j])) * localPos

Obviously this suffers from collapsing matrices, since matrices cannot be blended like this, but that's what the compromise the industry takes (dual quaternion aka geometric algebra skinning solves this, but glTF didn't standardize that yet)

So again, I don't see the skeleton root in any of these formula's, so I must be missing something.

Disclaimer: I haven't tested any of the above formulas in code! :wink:

[Selmar]

@Raveler I'm not an expert either, but I did implement the gltf skin processing in our engine (mostly with exports from the babylon exporter).

1. I don't even use the skeleton root property. I simply traverse the hierarchy until I find a shared node for all the joints. That is my skeleton root. This seems to be the most reliably way to go about it.

2. That scenario seems invalid to me, as your joints are not part of the scene.

3. As pointed out by @Ziriax, the joints decide what goes where. If the spec could tell you to 'change' the skeleton root, the only thing that could seem to do is change the hierarchy, and thus the joints' positions, and it's definitely not clear how that's supposed to work.

As you can read from the discussion in this thread, the skeleton root property from the gltf is to be ignored.

[Raveler]

And what do you use your calculated skeleton root for? You say you traverse the hierarchy but what for? What is the purpose of this skeleton root?

[Selmar]

In our case, it is the root bone for the bone hierarchy used by the skin, nothing more. The complete hierarchy between all the joints and this bone is part of the skeleton. The node which contains the skinned mesh data is the parent of this root node, so any animations higher up in the hierarchy are still applied.

[Raveler]

I still don't get it at all. What is the root bone relevant for? Practically, where in your code do you use it or refer to it?

[ziriax]

@Raveler So this seems purely related to the animation engine, so I would ignore this root skeleton property all together for now in our own engine. It seems that in other engines - like OGRE3D - the roots are only used when creating an instance for the skeleton, by walking the tree a single pass, but it is certainly possible to do that without. Also in OGRE3D, a skeleton has multiple roots, GLTF doesn't support that anyway.

If you need more info, these two threads might help, but certainly don't solve the issue completely. https://github.com/KhronosGroup/glTF-Blender-Exporter/issues/66 https://docs.unity3d.com/Manual/RootMotion.html

[Selmar]

We have a bone hierarchy that is separate from the node hierarchy. The skeleton root has no real practical use other than being the first node in the bone hierarchy. Either way, this is a bit off-topic indeed. Sorry.

[scurest]

If the skeleton property is present, can I assume it is a (possibly improper) ancestor of the joints? I'm using it in the Blender importer as a hint for where to place the armature.

[ziriax]

@scurest I think it is safe to assume that yes, but only the authors of the glTF spec can confirm this...

[shrinktofit]

@Ziriax I have the same question --- I can't understand the effect of skin.skeleton.

[Selmar]

@shrinktofit As far as I know, the skin.skeleton property may be ignored. Judging from the different discussions I have seen so far it may be removed eventually.

I guess there is one unclear use case: when a skin references a joint that is not part of the scene. In this case its transform should be relative to either the scene or this skeleton root. I would go with the scene root.. (In our engine we currently don't support this use case, though.)

[donmccurdy]

I don't believe skin.skeleton should affect calculation of the joint matrix. There are cases where it can be helpful, but none (that I know of) where it is strictly necessary. There may be more discussion needed to confirm that, but if it is the case, then the following language in the spec is incorrect:

Client implementations should apply only the transform of the skeleton root node to the skinned mesh...

If there were substantial complexity or lack of engine support for cases this implicitly enables, examples would be appreciated.

I guess there is one unclear use case: when a skin references a joint that is not part of the scene.

See example (3) in https://github.com/KhronosGroup/glTF/issues/1403. This case isn't well supported today; I think there may be an argument to disallow it. Have you seen this case occur?

[Selmar]

No, but we exclusively use the babylon exporter. I am not the best person to ask this.

I do believe it should be disallowed. It complicates traversing the node hierarchy in a pre-pass. If it is referenced by a skin, it is effectively part of the hierarchy anyway. Why do so implicitly?

The only useful scenario I can think of is when several skins inside the same scene reference the same joint with the intention of having its transform be relative to the skin itself (or to the skeleton root). However, is this even theoretically possible in current 3d modeling applications? Using 3dsmax's instancing, for example? And even in this scenario, the only thing it really does is prevent some node and animation data duplication. While this sounds nice, I don't believe it would be used a lot as it complicates exporter and importer logic.

[donmccurdy]

See additions in https://github.com/KhronosGroup/glTF/pull/1552, and two insights following from #1403 –

1. All joints should be members of the same scene hierarchy as the node in which that skin is used.
2. Joint hierarchies that do not have a common root (regardless of whether a skeleton property actually marks that common root) present an obstacle for import to DCC tools.

The existence of a root joint has valuable for portability across tools. Although the skeleton property is optional, its correct use is sufficient to ensure that portability:

The skeleton property (if present) points to the node that is the common root of a joints hierarchy or to a direct or indirect parent node of the common root.

Perhaps skeleton should become required in a future version, or perhaps the existence of a common root should be required without actually keeping a skeleton property. But I think that I would prefer not to make the spec less strict than it is now.

[lexaknyazev]

perhaps the existence of a common root should be required

It's already required by #1552.

[donmccurdy]

Oh thanks, I missed that its requirement was independent of the presence of the skeleton property. In that case, I think this issue is resolved?

[lexaknyazev]

I'd say yes.

[shrinktofit]

@lexaknyazev I still cannot understand how does the skeleton property affect the calculation of the skinning.

[donmccurdy]

It does not factor into the math of the skinning calculation. The skeleton property is a hint for tools that assign particular meaning to the root of the skeleton, and ensures that glTF models are compatible with such tools.

[Ybalrid]

The skeleton property is a hint for tools that assign particular meaning to the root of the skeleton, and ensures that glTF models are compatible with such tools.

In case such tool loads a glTF that doesn't specify this property (as this is a valid case because it's optional) what is the correct behavior?

Parsing the node hierarchy until a common ancestor node for all the joints specified in the skin is found and setting this one as the root of the skeleton?

[donmccurdy]

I think that depends on the tool, and what it needs to do with the skeleton root. In some cases just using the scene root may be fine, but what you describe would also work.