SkeletonUtils `.retarget()` and `.retargetClip()` error with documented `SkeletonHelper` params

[mattrossman]

Description

SkeletonUtils Docs indicate that .retarget() and .retargetClip() expect two SkeletonHelper as the source and target.

When I call retarget() with these arguments, I get an error:

❌ SkeletonUtils.js:31 Uncaught TypeError: Cannot read properties of undefined (reading 'bones')

The traceback points here:

https://github.com/mrdoob/three.js/blob/a55464eacb39e9a781a91d7848eb7401621d1045/examples/jsm/utils/SkeletonUtils.js#L31-L32

source.isObject3D reads true for a SkeletonHelper, so it then tries to read source.skeleton.bones which is invalid for SkeletonHelper. Instead, bones are stored on SkeletonHelper.bones.

I've tried to piece together what type(s) retarget is meant to accept. Taking a look at that getBones function:

https://github.com/mrdoob/three.js/blob/a55464eacb39e9a781a91d7848eb7401621d1045/examples/jsm/utils/SkeletonUtils.js#L485-L489

My guess is that retarget is written to accept:

• SkinnedMesh (according to source.skeleton.bones)
• Bone[] (according to Array.isArray in getBones)
• Skeleton (according to skeleton.bones in getBones)

I tried passing my SkeletonHelper.bones array instead, but that didn't seem to perform retargeting. Then I tried passing a SkinnedMesh and it sort of worked, but the retargeted pose was wrong. Then I tried passing a Skeleton and that worked as expected.

I notice in the Fiddle shared in #25288, they had to explicitly assign a skeleton to skeletonHelper.skeleton .

I get the impression that retarget() and retargetClip() are generally outdated, either in implementation or documentation. As @sunag mentioned in #25589 this could use an official example. I'd also like there to be some explanation of the options object.

Reproduction steps

1. Load a humanoid animation and create a SkeletonHelper helperSource for it
2. Load a humanoid character and create a SkeletonHelper helperTarget for it
3. Call SkeletonUtils.retarget(helperTarget, helperSource, {})

Code

const gltfSource = await loadGLTF(urlSource)
const helperSource = new THREE.SkeletonHelper(gltfSource.scene)

const gltfTarget = await loadGLTF(urlTarget)
const helperTarget = new THREE.SkeletonHelper(gltfTarget.scene)

SkeletonUtils.retarget(helperTarget, helperSource, {})

Live example

https://jsfiddle.net/mattrossman/h2473jf0/4/

Screenshots

No response

Version

r151

Device

Desktop

Browser

Chrome

OS

MacOS

[Mugen87]

I get the impression that retarget() and retargetClip() are generally outdated, either in implementation or documentation.

The retarget methods were added long time ago in context of Sea3D. When the related demos have been removed, both methods ended up without code examples.

[mattrossman]

I'm willing to make contributions here. I'm thinking of:

• updating docs to show source/target as Skeleton type and attempt to explain options similar to how export options are displayed in DRACOExporter docs
• adding a retarget and/or retargetClip example (webgl_animation_skinning_retarget?)
• updating retarget/retargetClip implementation to focus on one source/target type (Skeleton)

[Mugen87]

That would be awesome! Regarding #25763, I really want to remove the skeletonHelper.skeleton hack that you have also encountered in this issue. So limiting the parameter type of retarget() and retargetClip() to a single type would be great. I just wonder if we should use SkinnedMesh instead of Skeleton since this is the object the user works usually on app level.

[sunag]

@mattrossman Thanks for the initiative, if you download the r108 version and browse the SEA3D BVH examples it is possible to see some examples, I don't know exactly what may have changed since then but I think they will help.

https://github.com/mrdoob/three.js/releases/tag/r108

[mattrossman]

I just wonder if we should use SkinnedMesh instead of Skeleton since this is the object the user works usually on app level.

This is a good point to consider. I wonder what kind of use cases others have. Personally, my use case for retargeting is similar to the SEA3D example. My source is a Mixamo skeletal animation (without a mesh) and target is a skinned character mesh. So the most ergonomic combo for me is Skeleton source and SkinnedMesh or Skeleton target.

One benefit of the Skeleton type is if users are working with a SkinnedMesh, it's trivial for them to access the Skeleton via the .skeleton property whereas if they have a bare Skeleton animation, they'd need to make a dummy SkinnedMesh as shown in #25763. IMO it's not the most intuitive, until that PR I'd always thought a SkinnedMesh needs a "mesh" based on the name. Also, many of the SkeletonUtils functions use Skeleton inputs already.

That being said I need to take a deeper look at the retarget implementation. I don't fully understand all the options but I get a sense that some of them may rely on Object3D behavior, at least for the target. For instance, these lines where it uses target.matrixWorld

https://github.com/mrdoob/three.js/blob/a55464eacb39e9a781a91d7848eb7401621d1045/examples/jsm/utils/SkeletonUtils.js#L63-L79

[mattrossman]

Notes as I try understanding the usage of the existing options for documentation and code cleanup.

Options for retarget():

• hip - name of the hip bone in the source skeleton
• names - dictionary that maps from target bone names to source bone names
• preserveHipPosition - if enabled, preserves the Y component of the target hip bone position and zeros its X and Z component
  • I assume this was intended to be used alongside some root motion logic but I feel like not useful in its current state.
• preserveMatrix - ?
  • The implementation doesn't appear to be preserving matrix data, confused what this is supposed to do
• preservePosition - if enabled, preserves the original .position of target bones except the hip
  • Why this is useful? The only bone that I'd expect to have a .position track is the hip, the others only rotate.
• useTargetMatrix - if enabled, this will maintain the impact of target.matrixWorld when calculating target bone positions (defaults to disabled, in which case the inverse of target.matrixWorld will be applied)
  • Seems like the default is what you'd always want so that you can retarget even if your target isn't at the origin.

Options for retargetClip() (all options from retarget() are also valid here):

• useFirstFramePosition - if enabled, applies a negative offset to the target hip bone based on its retargeted position on the first frame. This effectively "centers" the retargeted animation at the origin.
  • Is this needed as an option? Seems like something that users could handle themselves.
• fps - determines rate at which animation will be sampled
• names - same purpose as the names option from retarget()
  • Note that here it defaults to an empty array, seems like this is supposed to be an empty object

In the Sea3D examples code:

• webgl_loader_sea3d_bvh
• webgl_loader_sea3d_bvh_retarget

I see usage of hip, names, and preserveHipPosition options from retarget() and useFirstFramePosition option from retargetClip(). The preserveHipPosition usage doesn't count since it's using the default of false, and although useFirstFramePosition is used here I suspect this is only useful for that particular SEA3D asset. I don't see usage of the other options.

In practice, I can get expected retargeting results using only hip and names. I'm inclined to remove the other options from retarget unless we have a clear use case for them.

[sunag]

preservePosition -> It can be useful if the source and the target have a different anatomy or size, in which case it preserves the original position of the bones, which physically would be the most correct but as we have exceptions like the character Dalshin from Street Fighter for example who stretches his arms would be It is important to have this option of choice.

useFirstFramePosition -> this creates an offset for the position resetting it, this can be useful mainly when using mocap that were not treated, some occasions it is better that the delta position is done in programming than by animation, for these cases it can be useful.

[timbotimbo]

@mattrossman I created a new example for my retargeting issue in 25288. This retargets the files from existing loader examples for a working animation. pirouette.bvh onto the model from Samba Dancing.fbx. Maybe this retarget with up-to-date example files can help in creating a full example.

jsFiddle

I tried the same with GLB models (soldier & xbot), but I can't get the scale working for those. They get scaled x100 when added to the scene, but retargeting scales them back down to a tiny size. Applying the scale again after the retarget messes with the root motion of the animation.

[mattrossman]

@timbotimbo Thank you for the example! I had similar scaling difficulties when trying the soldier & xbot models. That would be a good litmus test for the retargeting implementation, to make sure it can work for those assets as expected. I have used retargeting on my own GLB characters and Mixamo animations in a different project successfully so there must be a way to get those working.

@sunag Ok, I see how useFirstFramePosition can be useful for untreated mocap data. So far I have been using Mixamo animations which are pretty clean. Maybe a different name for this option could more clearly communicate how it centers the animation. To me, this name sounds like it preserves the first frame position in the result, but it does the opposite.

I don't fully understand preservePosition yet. In your example, would Dalshin be the source animation or the target character? I understand basic retargeting to work like so:

• align world position of root bone (hips) only, from source to target
• align world orientation of all bones, from source to target

Therefore, I would expect the target character's proportions (defined by the positions of bones) to remain untouched by retargeting (i.e. "preserved") without needing to .clone() them as I see in the current implementation. Only the hip bone's position is affected. Similarly, if the target's bones change proportions (e.g. Dalshin stretching arms), that would work too since these non-hip bone positions aren't affected by retargeting?

If however, Dalshin is the source animation, then I could see how a way to opt-in to transferring positions of bones other than the hip is valuable. However, transferring world positions 1:1 isn't what I'd want, because this would overwrite my target character's body proportions. Instead I suppose I'd want to apply the position deltas from the source animation (e.g. deltas on Dalshin's arm positions). It sounds like that'd be a responsibility of retargetClip instead of retarget, since it could calculate those deltas.

[mattrossman]

Here is the result I get when retargeting that pirouette BVH animation to the Soldier model: https://jsfiddle.net/mattrossman/byhm96dp/1/

I get the same result with preservePositions: true in the retarget options. Most of the options I try don't seem to have an effect.

One difficulty in debugging this sort of thing is that THREE.SkeletonHelper is a bit of a red herring. It only shows the position of bones, not the orientations. It would be easier to understand how it's behaving with a visualization like this with the axes of each bone.

Normally I'd add a THREE.AxesHelper to each bone myself, though with the dummy SkinnedMesh pattern from #25763 that's difficult because .visible = false to prevent an error, so my axes also get hidden. So, I have to use the helper.skeleton = skeleton hack instead.

Here's a copy that shows the axes for the source and target bones: https://jsfiddle.net/mattrossman/byhm96dp/2/

You can see that the models use different bone orientations in their T-pose (the Soldier has the +Y axis pointing along the bone chain). Maybe this could contribute to the bones getting twisted up. I don't remember if this implementation retargets orientations relative to the bind pose or not. Not sure why their position and scale is being affected though. The BVH clip doesn't contain any scale tracks for other bones. Besides, I feel that by default retargeting shouldn't mess with scale or position of non-hip bones.

[alankent]

I was trying to get the retargeting to work and came across this very helpful thread. I am trying to do it using Fiber (react wrapper around three), so trying to work out how some of the examples map across. I have a few outstanding problems

• Some of my models have a root bone (I want them in general) but there is no root animation preserved that I can see (e.g. for walking you move the root x/z values)
• Playing a mixamo walk animation directly on the mixamo model works. Retargeting the animation clip to itself (!) makes things move, but the character is doubled over and not moving forward (no root motion).

Before retargeted animation is applied:

While playing retargeted animation:

Clearly I am doing something wrong, I just thought I would check in to see if this thread had made any progress on new samples etc. before I keep investigating. I was trying to retarget to other characters with root bones, but when I retargeted it to itself I realized that was not even working, so I am ignoring the root bone issue for now. (Normally you want forward root motion on the root bone, making it easier to turn off later without messing up the rest of the animation etc.)

Thanks!

[mattrossman]

@alankent My main finding after lots of trial and error was that Three's example retargeting function is very sensitive to things like differences in local bone transforms, difference in bind pose bone orientations, timing of matrix updates. I ended up writing my own retargeting logic for my project rather than try to work around it. I assume models are bound from a T-pose in my solution and rely more on world space transforms to account for rig differences.

I was hoping to translate some of these findings into a new .retarget() example for Three, but there's so many different options that affect the final result or introduce performance implications. I don't know how to make a "one size fits all" retargeting method.

By comparison, I found Unreal Engine has a much more sophisticated animation system that provides a suite of animation primitives that make retargeting easier. I know @sketchpunklabs is doing some interesting work in a similar regard with ossos, that might be a library to check out in the meantime for your retargeting needs.

[alankent]

Thanks for the reference. I am having a look at ossos now. I have some concerns around how easy it will be to merge into my existing app, but maybe that is just lack of familiarity. He has lots of YouTube videos to back it up, has IK support, and spring bones built in. So lots of nice stuff. Time to start exploring the demos that come with it!

[alankent]

In case useful to anyone else on the same journey, I came across https://pixiv.github.io/three-vrm/packages/three-vrm/examples/humanoidAnimation/loadMixamoAnimation.js which loads a mixamo.com animation clip on a VRM character, doing retargeting. It is hard coded to particular bone names, but very useful as reference code.

[trusktr]

Hello folks, I started a thread about this in the forum before I saw this one, showing some live examples of transform issues:

https://discourse.threejs.org/t/fixing-skeletonutils-retarget-and-retargetclip-functions/65149

The main problem to work out is transform handling (fixing parameters and .skeleton hack is comparatively easier).