Implement Dual Quaternion Skinning AKA "Preserve Volume" option compatibility with Blender

[beiller]

Describe the project you are working on

Working on a character importing and posing application where the user can use VR to import / export poses and animations.

Describe the problem or limitation you are having in your project

There is currently no compatibility in Godot with blenders "preserve volume" option on Skeletons

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With some small changes to the Skeleton3D pipeline Dual Quaternion can be implemented and offer better compatibility with Blender.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Initial implementation PR is here: https://github.com/godotengine/godot/pull/89131

• Dual Quaternion transform needs to be generated from the matrix transform per-bone.
• 2 Quaternions need to be passed into the vertex() program, and blended together (x4 or x8)
• The final transform is computed in vertex() program in mat4 format and applied to the vertex

If this enhancement will not be used often, can it be worked around with a few lines of script?

This can be implemented in godot via gdscript / shader. That would require the complete implementation of BasicMaterial in glsl and is not easy to carry on through version changes etc.

Is there a reason why this should be core and not an add-on in the asset library?

As above, it requires very significant effort to write a ShaderMaterial and to keep that material in line with existing BasicMaterial seems to be very difficult / impossible, and the hypothetical add-on written would not be compatible with any minor changes in the engine going forward. The change as presented (in-core) is much more portable and should survive major refactors.

[fire]

If I remember, @reduz was not convinced by the use of Dual Quaternions for animations. Maybe something has changed.

Personally I think exporting dual quaternions requires a complicated baking procedure to export as gltf.

[beiller]

Hey its not correct see the linked PR for a full working implementation. It just involves converting the bone transform mat4 to 2 quaternions, passing those quaternions to the existing shader, and in the shader blending the quaternions and converting back to matrix 4. No data changes at all required.

[clayjohn]

Hey its not correct see the linked PR for a full working implementation. It just involves converting the bone transform mat4 to 2 quaternions, passing those quaternions to the existing shader, and in the shader blending the quaternions and converting back to matrix 4. No data changes at all required.

Is it correct to say that the benefit of this approach is just the lowered bandwidth requirement? And the tradeoff is increased instruction count both inside the shader and on CPU?

[beiller]

Hey @clayjohn the benefit of dual quaternion transforms for Skeletons is that twisting a skeletal joint causes the mesh to collapse under linear transformation. Using the quaternion blending "preserves the volume" during rotation. You can see the difference in blender using an armature and bending it like a knee joint and toggle on/off "preserve volume". In linear space the faces will collapse into very thin areas and the DQ transform will preserve the volume of the joint. It's used in a lot of character rigs for joints for that reason.

edit: reference http://rodolphe-vaillant.fr/entry/29/dual-quaternions-skinning-tutorial-and-c-codes

[TokageItLab]

When a skin is linearly deformed, the large curvatures cause the skin to lose its shape. This is usually solved with BoneConstraint and subsidiary bones, but Dual Quaternion serves as another solution. BoneConstraint is still a work in progress and will be implemented as the next step after SkeletonModifier.

Dual Quaternion will improve the appearance of skinning in some cases, although it will be twice as computationally intensive since two Quaternions will be needed for each calculation.

The problem is that the skin will not thin, but on the other hand, it may expand unnaturally, making it unsuitable for low-polygon model or hard surface models. Also, as mentioned above, many animations are resolved with auxiliary bones usually, so this feature is not required always. Whether this is necessary or not is on a case-by-case so may be useful in animations with a pixar & disney looks.

I am not opposed to including this feature in core because it would be difficult to make it an extension, but if it is implemented, it needs to be optional per MeshInstance. Also, a warning should be given that this skinning setting is generally ignored in glTF Export, as it is platform-dependent whether this calculation is done for skinning by similar reason for NLA animation.

[beiller]

@TokageItLab nice thanks for the writeup. Low poly definitely sounds like it could cause some problems. Not sure how bad they would be, but on my TODO in the PR is a toggle it should not be difficult to implement on Skeleton3D object.

For the computation modern graphics card has no issue. The technique seems to come from around 2006. I think theoretically the bandwidth savings is worth more than computation cost. But we currently do not even take advantage of lower bandwidth since in my current PR the quaternions are packed into the matrix.

[TokageItLab]

Not sure how bad they would be, but on my TODO in the PR is a toggle it should not be difficult to implement on Skeleton3D object.

A Skeleton may have multiple MeshInstances. For example, if you want skinning methods to coexist on the rubber skin mesh (Dual Quaternion skinning) and armor parts of a hard-surface mesh (Linear skinning) for robot which bound same Skeleton, I think it would be more preferable to have a toggle on the MeshInstance side. Ideally, it should be allowed to be changed in the import settings as well.

BTW, since there are several other possible methods of skinning besides DualQuaternion, it would be desirable if the implementation way could be extended to them in the future. Well, boolean is fine for now even if we make it an enum in the future. So it is fine if the calculation methods can be cleanly separated between linear blending and dual quaternion blending. In other words, we need to avoid hard-coding for dual quaternion blending, define the functions for linear blending and dual quaternion blending in the same way, and call it depending on the option.

Real-time Skeletal Skinning with Optimized Centers of Rotation - Youtube

[fire]

For example, a newer technique is called direct delta mush https://github.com/V-Sekai/unity-mesh-deform-direct-delta-mush and I agree with @TokageItLab that the code can be worth abstracting between linear blending and dual quaternion blending.

[beiller]

I think that to accomplish these other things, the vertex pipeline just needs to be programmable in Godot similar to shaders. Currently the vertex pipeline is a compute shader but it can't be swapped out in the editor. That should be the future direction imo.

Setting dual quaternion use per mesh instance is tricky as well it means we have to be less efficient and compute both dual quaternion and linear blend matrices always or said another way, all skeleton has to compute dual quaternion. Which maybe isn't a big deal.

[TokageItLab]

I think the performance loss is simply a matter of architectural design and can be avoided by using a cache. If there is at least one DQS, the calculation result array for the DQS can be cached and used for multiple MeshInstances, the same goes for LinearSkinning. In other words, if both Skinning coexist/be used by MeshInstances, there will be two caches, and if only one of them is used, there will be one cache.

[octanejohn]

related i think

https://github.com/enkimute/LookMaNoMatrices

https://enkimute.github.io/LookMaNoMatrices/

[firebeetleLobster]

Try this: https://godotengine.org/asset-library/asset/2062

[computerologist]

BTW, since there are several other possible methods of skinning besides DualQuaternion, it would be desirable if the implementation way could be extended to them in the future. Well, boolean is fine for now even if we make it an enum in the future.

Superior to this would be a weighted blend of techniques, per vertex. It is inherently inefficient, but allows artists the option to trade efficiency for maximally localised quality across a mesh--the typical example of this is in the DQS vs LBS optimizations of candywrappering of forearms vs inflation of elbows.

[firebeetleLobster]

It is in fact possible to realize the dual quaternion skinning method with a spatial shader. Despite the inefficiency, it works.

[beiller]

Yes I tried this approach. When you write a custom shader now all the spatial shader techniques are gone unless you rewrite them yourself. It's very difficult to use in practice I found.