Suggestion: Support animating multiple SkinnedMeshes using a single Skeleton

[fallenoak]

Description of the problem

According to this PR, https://github.com/mrdoob/three.js/pull/4812#issuecomment-43788375, there is ostensibly a way to support animating multiple SkinnedMeshes using a single Skeleton:

Please see the most recent code. I have re-introduced THREE.Skeleton and decoupled it from THREE.SkinnedMesh. This will allow us to animate multiple meshes with a single skeleton. The mesh-specific bind matrices are stored in THREE.SkinnedMesh and used by the vertex shader to transform to/from "bind space" (world space in the current implementation).

Unfortunately, no docs or examples appear to have been contributed to demonstrate how this might be accomplished, and try as I might, I can't seem to come up with a way to achieve this on my own.

I'd like to suggest either adding an example, updating the docs, or both. Alternatively, in the event that it's not currently possible, does anyone have a suggestion on an approach to make it possible in the future?

Three.js version

• [ ] Dev
• [x] r80
• [ ] ...

  Browser

• [x] All of them
• [ ] Chrome
• [ ] Firefox
• [ ] Internet Explorer

  OS

• [x] All of them
• [ ] Windows
• [ ] Linux
• [ ] Android
• [ ] IOS

[titansoftime]

I'm right there with you. I am about to start posting some detailed inquiries regarding how to most efficiently animate multiple meshes with the new animation system (some share geometry, some do not).

[satori99]

@titansoftime / @fallenoak I have achieved this (multiple meshes sharing same skeleton) using the current dev branch, but its a bit of a pain to do with SkinnedMesh and BufferGeometry currently. First, I had to modify the ObjectLoader to load SkinnedMeshs without bones from JSON properly.

Then, I used a Group instance as a stand-in for an armature root, with multiple skinned mesh children. And I stored the shared bones in a custom userData property in the JSON, so it survives the loading process.

You will have to handle the Bone and Skeleton creation manually in your code. These can be attached to the Group root too. Then binding that Skeleton to each skinned mesh is straightforward. The bone and skeleton creation code is more or less the same as what the SkinnedMesh constructor usually does when bones are present on the mesh geometry.

[fallenoak]

@satori99 Hrm, thanks for the details! I'm already working directly with SkinnedMesh / Skeleton / Bone (ie not using ObjectLoader) because the model formats I'm using come from a third party and are non-standard.

So if I understand things right, you're adding the Bones to a Group, placing that Group in the scene, and then spawning multiple SkinnedMeshes, each sharing the same Skeleton (and thus the same Bones across each SkinnedMesh). Is that right?

Does it matter where in the scene the Group that holds the Bones goes? Is the Group's placement irrelevant, as long as it's part of the scene graph? Does it have to all of the SkinnedMeshes that are binding to the Skeleton?

What happens when the matrixWorld of the SkinnedMeshes changes after calling bind()? It looks to me like bind() copies the matrixWorld when it's called, but how does that deal with cases like a translating / rotating / scaling SkinnedMesh?

Relevant code for bind():

    bind: function( skeleton, bindMatrix ) {

        this.skeleton = skeleton;

        if ( bindMatrix === undefined ) {

            this.updateMatrixWorld( true );

            this.skeleton.calculateInverses();

            bindMatrix = this.matrixWorld;

        }

        this.bindMatrix.copy( bindMatrix );
        this.bindMatrixInverse.getInverse( bindMatrix );

    }

[satori99]

So if I understand things right, you're adding the Bones to a Group, placing that Group in the scene, and then spawning multiple SkinnedMeshes, each sharing the same Skeleton (and thus the same Bones across each SkinnedMesh). Is that right?

I am loading SkinnedMesh instances directly from JSON, so the loader is creating them. But I am loading the skinned meshes without any bones on them (so the SkinnedMesh constructor code does nothing really). Creating the SkinnedMesh instances yourself shouldnt make any difference though. All the SkinnedMeshs plus the Bones I created manually are all parented to the same Group. You can translate rotate the Group to move the whole character in the scene.

Is the Group's placement irrelevant, as long as it's part of the scene graph?

I believe so.

Does it have to all of the SkinnedMeshes that are binding to the Skeleton?

Not if you don't want to; Any static mesh or a skinned mesh with its own skeleton should do its own thing still, relative to the parent. When I apply animationClip's, I use the Group instance as the mixer root object. I have not encountered any issues doing that so far.

Ill try to post a fiddle demo later today.

[satori99]

This is demonstration of the technique i am using to share a skeleton between multiple SkinnedMesh instances:

https://jsfiddle.net/satori99/pay0oqcd/

The Body and Clothing are separate SkinnedMeshs. A skeletal animation is applied to the root Group instance (just one leg). The whole thing is then animated in the render loop independent of the skeletal animation.

[fallenoak]

@satori99 So your example isn't quite the same as what I had in mind, unfortunately. Your method works well when handling meshes that are effectively segments of a single mesh, but doesn't cover the case where each SkinnedMesh sharing the Skeleton has a different placement (translation, rotation, scale) in the scene.

If you modify your example like so:

    body = createSkinnedMesh( root.getObjectByName( 'Body' ), skeleton )
    body.position.x = -2;
    body.position.y = 2;
    body.updateMatrix();
    body.updateMatrixWorld();

You'll find that the body is still in the same position. As far as I can tell, there's no way to offset the body and clothes SkinnedMeshes from one another. Their placement is locked to the position of the root.

Maybe sharing a Skeleton across submeshes was the only kind of sharing intended by @ikerr in his improvements, but it seems like it ought to be mathematically possible to share a skeleton across multiple scene placements.

[ikerr]

It should be possible to animate multiple SkinnedMesh instances with a single Skeleton. And yes, I should create an example for this... I'm away for the next week or so, but will try to get something working after that. Please give me a nudge if I don't.

The process would be as follows:

1. Create two THREE.SkinnedMesh instances.
2. Create a single THREE.Skeleton instance.
3. For each THREE.SkinnedMesh determine the "bind space matrix" that "attaches" the mesh to the skeleton. For example, if your mesh is at the origin, but your skeleton is at +1 along the x-axis, the bind shape matrix would be a (1, 0, 0) translation matrix.
4. Pass the bind space matrix to THREE.SkinnedMesh.prototype.bind(), along with the skeleton.

We should probably change the name of THREE.Skeleton to THREE.Skin since it's equivalent to FBX's FbxSkin and COLLADA's .

[fallenoak]

@ikerr Thanks for popping in.

I'm still foggy on how the bind space matrix would account for separate instances / placements of THREE.SkinnedMesh in a scene.

I believe I understand how to recycle a set of bones and skeleton for several THREE.SkinnedMesh instances that co-exist in the same local space--for example, skinned meshes acting as sections of a single parent geometry. But once the THREE.SkinnedMeshes are separate objects in a THREE.Scene(), my understanding goes out the window.

For example:

var scene = new THREE.Scene();

var bones = <an array of bones meant to be shared (not cloned) by multiple skinned mesh instances>;
var geometry = <a buffer geometry meant to be shared (not cloned) by multiple skinned mesh instances>;
var material = <a basic material with skinning set to true>;

var skeleton = new THREE.Skeleton(bones);

var object1 = new THREE.SkinnedMesh(geometry, material);
object1.bind(skeleton);
object1.position.x = 5;
object1.position.y = 5;
scene.add(object1);

var object2 = new THREE.SkinnedMesh(geometry, material);
object2.bind(skeleton);
object2.position.x = 10;
object2.position.y = 10;
scene.add(object2);

1. Where in the THREE.Scene should the bones be added? Do they need to go into the scene at all if I were to manually call updateMatrix() on them while they animate?
2. What, precisely, is the purpose of bindMatrix and bindMatrixInverse? If my bone positions are in the same coordinate system as the geometry, should I just be using a bindMatrix set to identity, and a bindMode set to detached?
3. Given that object1 and object2 would have regularly updated position, scale, and rotation independent of each other, if I need to use something other than identity, how could I calculate an appropriate bind space matrix for them?

[fallenoak]

To build a bit on my last comment, it appears that it's possible to animate multiple independently positioned THREE.SkinnedMesh objects by using an identity matrix for the bindMatrix and using bindMode set to detached.

When I do this, I'm able to animate the THREE.Bone objects just once, and share the results between multiple THREE.SkinnedMesh placements of the same geometry in the scene graph.

I'm still not clear on why this works, but it'd be lovely to update the skinning documentation with a few more details, and add some examples.

Nudging @ikerr!

[mrdoob]

We probably should do an example showing this 😅

[takahirox]

I want such an example, too.

I've been working on SkinnedMesh serialization #8978. I wanna consider how to support shared Skeleton on that. I need a standard way to share Skeleton.

[fallenoak]

Nudge @ikerr and/or @mrdoob

[ikerr]

Started working on this, but am getting stuck using the Blender exporter. Maybe you guys can help me out...

I'm trying to export a skinned, animated model and would like to use the "new" animation format (with the "tracks" property), but no matter which combination of settings I use, the "animations" property always ends up looking like this:

"animations":[{                                                             
    "fps":24,                                                               
    "name":"default",                                                       
    "tracks":[]                                                             
}], 

I'm deselect everything before exporting and use the following export settings:

Any ideas?

[mrdoob]

/ping @bhouston @tschw

[bhouston]

I'd suggest that someone create an example that demonstrates multiple skinned meshes being animated using a skeleton. If you can not get it working, that would be a useful test case, and then once it is working, we can maintain it going forward so that it always works.

[ikerr]

Hi @bhouston, I'm working on such an example but am having trouble with the Blender exporter. Please see my previous comment for details. Do you know what settings I need to export the "new" animations?

[bhouston]

@ikerr I haven't tried the latest Blender exporter in a couple months. It was exported skinned animations with skeletons earlier this year. But it could also be that your scene is setup differently than the ones I was testing with. I would suggest trying the Blender importer from earlier this year -- like February and see if it works. If it doesn't then it may be that your scene structure is not supported by the plugin.

[yrns]

@ikerr Without the blend file it's hard to say. Currently I'm able to export skinned animations from Blender 2.78 and a recent exporter from 32867de625840be971a47f7f07c25aacd448784c.

Edit: Sorry, I'm also not getting "tracks", just the old style hierachies with type Geometry.

[milcktoast]

I think part of the problem is problematic coupling in the constructor of SkinnedMesh which makes users jump through some hoops to allow sharing of bones / skeleton. One possible solution is to define Skeleton as a completely separate object in the scene graph which can share its bone data with any direct descendent children which are SkinnedMesh instances. This is similar to how Blender implements skinning, with Armature objects as separate from Mesh.

[milcktoast]

@mrdoob @ikerr @empaempa As far as I can tell, skin property of Bone is no longer needed. I'm not sure the original intention of this property, but I couldn't find a reference to it in the renderer or other source code. Removing this relationship would further help to simplify how Skeleton and Bone relate to SkinnedMesh instances.

[mrdoob]

@mrdoob @ikerr @empaempa As far as I can tell, skin property of Bone is no longer needed. I'm not sure the original intention of this property, but I couldn't find a reference to it in the renderer or other source code. Removing this relationship would further help to simplify how Skeleton and Bone relate to SkinnedMesh instances.

Try doing a PR and see if nothing breaks 😊

[YannisJ]

Hello,

Is there anything new or planned about multiples meshes for one skeleton ? I looking to do that in a futur project but apparently there is no example yet (except for the satori99's proposition)

[godknowsiamgood]

+1 still not sure how to properly implement person with clothing with same skinning animation.

[funwithtriangles]

I'm tackling this issue right now. Will report back if I get an example working, otherwise it would be great to know if anyone else has had success?

[titansoftime]

I personally am waiting for the Khronos Group GLTF2.0 exporter for Blender to work out their animation bugs before doing any heavy animation stuff.

[funwithtriangles]

@titansoftime it's a real pain, I know! Just so you know I wrote a little guide helping with this topic (basically the best option right now is to convert from FBX to glTF).

Currently seeing if I can put together a simple example for shared skeletons. Will submit a PR if it is successful :)

[funwithtriangles]

Example added. Open to feedback, even if it's just more helpful comments, etc. :)

https://github.com/mrdoob/three.js/pull/16608

[Jack-Carbone]

I was able to export each skinned mesh with the same armature, and then set all skeletons but one equal to just a single skeleton to get them all to follow the same pose. I sort of tricked the engine into already having weights on each mesh, and then sharing a single skeleton.

[danielefederico]

Hello there, is there any news about this issue? The original problem was how to support animating multiple SkinnedMeshes using a single Skeleton.

As for @fallenoak , I have more than one mesh being driven by the same animated skeleton, but it's clearly not working as the skin weights look all messed up.

Thanks, Daniele

[Jack-Carbone]

Yes, I actually created my own copy of a class called something like InstancedSkinnedMesh from an older three js version and it worked flawlessly. I will try to reply with that file shortly.

[Jack-Carbone]

Hello,

I have all of the resources now. I am still using this implementation myself, but I haven't actually gotten around to using it for my project yet. My project is still not ready, but soon, I will need it again. Anyways, everything is shown below. Thanks!

Here is basic example usage in js:

Create New Instanced Skinned Mesh: const mesh = new InstancedSkinnedMesh.InstancedSkinnedMesh(geometry, material, count);

Set Usage If Wanted mesh.instanceMatrix.setUsage(THREE.DynamicDrawUsage);

Set Transformation Matrix By Copying a Beihind the Scenes Dummy to the Instanced Data mesh.setMatrixAt(count, dummy.matrix);

Here is the source file I half-made from an old version. I will also attach a file. Sorry, it is in js, but you can ask ChatGPT or something to convert it if you want a typed version easily. Code:

import { SkinnedMesh, InstancedBufferAttribute, Matrix4 } from 'three';

//Needed THREE MATRICES const _instanceLocalMatrix$1 = /@PURE/ new Matrix4(); const _instanceWorldMatrix$1 = /@PURE/ new Matrix4(); const _instanceIntersects$1 = [];

export class InstancedSkinnedMesh extends SkinnedMesh {

constructor(geometry, material, count) {

      super(geometry, material);

      this.instanceMatrix = new InstancedBufferAttribute(new Float32Array(count * 16), 16);       this.instanceColor = null;       this.instanceBones = null;

      this.count = count;

      this.frustumCulled = false;

      this._mesh = null;

}

copy(source) {

      super.copy(source);

      if (source.isInstancedMesh) {

       this.instanceMatrix.copy(source.instanceMatrix);

       if (source.instanceColor !== null) this.instanceColor = source.instanceColor.clone();

       this.count = source.count;

      }

      return this;

}

getColorAt(index, color) {

      color.fromArray(this.instanceColor.array, index * 3);

}

getMatrixAt(index, matrix) {

      matrix.fromArray(this.instanceMatrix.array, index * 16);

}

raycast(raycaster, intersects) {

      const matrixWorld = this.matrixWorld;       const raycastTimes = this.count;

      if (this._mesh === null) {

       this._mesh = new SkinnedMesh(this.geometry, this.material);        this._mesh.copy(this);

      }

      const _mesh = this._mesh;

      if (_mesh.material === undefined) return;

      for (let instanceId = 0; instanceId < raycastTimes; instanceId++) {

       // calculate the world matrix for each instance

       this.getMatrixAt(instanceId, _instanceLocalMatrix$1);

       _instanceWorldMatrix$1.multiplyMatrices(matrixWorld, _instanceLocalMatrix$1);

       // the mesh represents this single instance

       _mesh.matrixWorld = _instanceWorldMatrix$1;

       _mesh.raycast(raycaster, _instanceIntersects$1);

       // process the result of raycast

       for (let i = 0, l = _instanceIntersects$1.length; i < l; i++) {

            const intersect = _instanceIntersects$1[i];             intersect.instanceId = instanceId;             intersect.object = this;             intersects.push(intersect);        }

       _instanceIntersects$1.length = 0;       } }

setColorAt(index, color) {

      if (this.instanceColor === null) {        this.instanceColor = new InstancedBufferAttribute(new Float32Array(this.instanceMatrix.count 3), 3);       }       color.toArray(this.instanceColor.array, index 3); }

setMatrixAt(index, matrix) {

      matrix.toArray(this.instanceMatrix.array, index * 16); }

setBonesAt(index, skeleton) {

      skeleton = skeleton || this.skeleton;       const size = skeleton.bones.length 16;       if (this.instanceBones === null) {        this.instanceBones = new Float32Array(size this.count);       }       skeleton.update(this.instanceBones, index); }

updateMorphTargets() {

}

dispose() {

      this.dispatchEvent({ type: 'dispose' }); } }

InstancedSkinnedMesh.prototype.isInstancedMesh = true;

[Jack-Carbone]

Everyone let me know if this works! We can close if so. It worked for me. I made a batch of hundreds and a batch of thousands of entities with one skinned mesh. It was fun!

[Mugen87]

There is now https://threejs.org/examples/webgl_animation_multiple that shows how to setup a shared skeleton 👍 .

[Jack-Carbone]

Isn't the shared skeleton solution still just using cloned meshes? I feel that this is not the same as the instanced skinned mesh issue that this thread is about. Technically, that is not a solution whereas mine is real GPU instancing of a skinned mesh.

[Mugen87]

Instancing in context of skinned meshes is discussed here: #25078

[danielefederico]

Unfortunately this is not what I'm looking for. I'm loading an FBX which contains multiple geometries all skinned to the same skeleton. When loading the FBX with the FBX loader I get a warning: "skeleton attached to more than one geometry is not supported." Then the animation looks funny in the viewport.

I'm not looking to use Instanced meshes, I just want different meshes to be affected by the same skeleton.

Just curious, is there any reason why this is not implemented?

Thanks, Daniele

[Jack-Carbone]

Gotcha, yeah sorry I am not sure. Alright well, good luck!

[Mugen87]

I'm not looking to use Instanced meshes, I just want different meshes to be affected by the same skeleton. Just curious, is there any reason why this is not implemented?

The example that I have linked in https://github.com/mrdoob/three.js/issues/9606#issuecomment-1889304517 actually does that. Multiple meshes affected by one skeleton. So it is technically possible.

I guess the issue is that FBXLoader does not make use of it yet.