SkinnedMesh: Implement skinning in a different coordinate space

jakezira commented 2 years ago

Environment

Code

Added midObj to set the offset.

<!DOCTYPE html>

<html>
    <head>
        <meta charset="utf-8" />
        <title>three-vrm example</title>
        <meta
            name="viewport"
            content="width=device-width, initial-scale=1.0, minimum-scale=1.0, maximum-scale=1.0, user-scalable=no"
        />
        <style>
            body {
                margin: 0;
            }
            canvas {
                display: block;
            }
        </style>
    </head>

    <body>
        <script src="https://unpkg.com/three@0.143.0/build/three.js"></script>
        <script src="https://unpkg.com/three@0.143.0/examples/js/loaders/GLTFLoader.js"></script>
        <script src="https://unpkg.com/three@0.143.0/examples/js/controls/OrbitControls.js"></script>
        <script>
            // renderer
            const renderer = new THREE.WebGLRenderer();
            renderer.setSize( window.innerWidth, window.innerHeight );
            renderer.setPixelRatio( window.devicePixelRatio );
            document.body.appendChild( renderer.domElement );

            const offset = 100000.0;

            // camera
            const camera = new THREE.PerspectiveCamera( 30.0, window.innerWidth / window.innerHeight, 0.1, 30 );
            camera.position.set( 0.0, offset, 5.0 );

            // camera controls
            const controls = new THREE.OrbitControls( camera, renderer.domElement );
            controls.screenSpacePanning = true;
            controls.target.set( 0.0, offset, 0.0 );
            controls.update();

            // scene
            const scene = new THREE.Scene();

            // light
            const light = new THREE.DirectionalLight( 0xffffff );
            light.position.set( 1.0, 1.0, 1.0 ).normalize();
            scene.add( light );

            // gltf
            const loader = new THREE.GLTFLoader();
            loader.crossOrigin = 'anonymous';

            loader.load(

                // URL of the glb you want to load
                './models/scilly_drophunter_v31.6_Guilty.glb',

                // called when the resource is loaded
                ( gltf ) => {

                    const midObj = new THREE.Object3D();
                    scene.add( midObj );
                    midObj.add( gltf.scene );

                    midObj.position.copy(new THREE.Vector3( 0, offset, 0 ));

                },

                // called while loading is progressing
                ( progress ) => console.log( 'Loading model...', 100.0 * ( progress.loaded / progress.total ), '%' ),

                // called when loading has errors
                ( error ) => console.error( error )

            );

            // helpers
            const gridHelper = new THREE.GridHelper( 10, 10 );
            scene.add( gridHelper );

            const axesHelper = new THREE.AxesHelper( 5 );
            scene.add( axesHelper );

            // update
            function animate() {

                requestAnimationFrame( animate );

                renderer.render( scene, camera );
            }

            animate();
        </script>
    </body>
</html>

Test 1: `offset=0` ==> Correct

Test 2: `offset=100000` ==> Bad rendering

donmccurdy commented 2 years ago

I think the issue remains that three.js implements skinning in world space, which means floating point precision is worse for positions much further from the origin. For now there is no workaround other than to avoid this scenario, e.g. by transforming the scene rather than the character.

WestLangley commented 2 years ago

Right. This is the same issue as #13288.

This is a three.js problem caused by skinning in world space.

The change to skinning in world space began in #4812:

Previously, we performed skinning in the local space of the SkinnedMesh. But now, since the objects/bones are not necessarily descendants, it is more natural to perform skinning in world space…

donmccurdy commented 2 years ago

Hm. If we believe that skinning should be performed in a local space, then which local space? I'm not sure that the local space of the SkinnedMesh would be the right choice. It's a common use case to transform the root Bone instead of the SkinnedMesh when moving a character, and the Bones may not be descendants of the SkinnedMesh.

I wonder if we could allow the user to define an Object3D as the 'root' (possibly the SkinnedMesh, possibly not) such that skinning would be performed in the local space of that Object3D. We could require that all objects used as bones be descendants of that node, and consider it invalid if this requirement is broken. If no such root is identified, the root would implicitly continue to be the Scene, i.e. skinning in world space.

glTF files have a relevant hint, a skin.skeleton node, that can optionally define a root of the joint hierarchy. I've always assumed that property was not relevant to three.js, but perhaps this is a good use case allowing us to guess the right local frame for a SkinnedMesh.

jakezira commented 2 years ago

We really need a workaround ... moving scene instead of avatar doesn't make sense, especially when you're relying on physx etc. We need a consistence game space. Also other objects don't have this problem. It's the implementation limit of skinned mesh of bone chain. Better to provide a solution on skinned mesh.

@donmccurdy

I wonder if we could allow the user to define an Object3D as the 'root' (possibly the SkinnedMesh, possibly not) such that skinning would be performed in the local space of that Object3D. We could require that all objects used as bones be descendants of that node, and consider it invalid if this requirement is broken. If no such root is identified, the root would implicitly continue to be the Scene, i.e. skinning in world space.

This is what I was going to implement. :grin: Looking forward for this fix.

Dadibom commented 2 years ago

We really need a workaround ... moving scene instead of avatar doesn't make sense, especially when you're relying on physx etc. We need a consistence game space. Also other objects don't have this problem. It's the implementation limit of skinned mesh of bone chain. Better to provide a solution on skinned mesh.

It is not that uncommon to offset the world space for better floating point precision though. This will not be your only problem when you are trying to run floating point engines at high coordinates.

autra commented 2 years ago

It is not that uncommon to offset the world space for better floating point precision though. This will not be your only problem when you are trying to run floating point engines at high coordinates.

We do at giro3d (because it's a geographic engine, dealing with coordinates in the millions sometimes). Yes there are other classes of problems, but so far they have been manageable without shifting the scene. Our use case is typically the same as @jakezira but worst. For instance, this is supposed to be the 3 Soldiers of https://threejs.org/examples/webgl_animation_multiple:

From what I understand the solution proposed by @jakezira should fix it for us as well.

sunag commented 2 years ago

Maybe a temporary solution can add an attribute vec3 globalPosition after skinning_vertex?

...
#include <skinning_vertex>
transformed += globalPosition;
...

I haven't tested it, but in my mint it should work...

donmccurdy commented 2 years ago

It may be the joints rather than the mesh that have a large displacement. In the case of VRM or glTF I think it's more likely the joints; skinned mesh displacement should be ignored.

mrdoob / three.js