SkinnedMesh culling is wrong and PointLightShadow gets clipped for SkinnedMesh

[gogiii]

Description of the problem

Not sure if this is the same bug or two different.

1. Whenever SkinnedMesh origin is out of camera frustum, the mesh gets culled even if its actual transformed geometry must be visible on the screen. Seems like the bounding box used for clipping is not updated with animation.

2. Same happens for shadows. When the SkinnedMesh moves away from its origin in light space, its shadow gets clipped/partially clipped by pointlight's perspectivecamera (I guess).

Here's an example in jsfiddle: https://jsfiddle.net/vtb6z50j/ Sorry for inline json. The example was dependent of viewport dimensions. Fixed.

And its not light.shadow.camera.far, light.shadow.radius bug. Actually the use case for this was to make a complete animation in any 3d editor (blender in my case) just to play it as is in three.js. So the character isn't moved by mesh.position, there is no need to sync movement/animation and hardcode movement positions. It just plays baked animation!

Update: yes, making frustumCulled = false does fix both problems (which is enough for me to continue), but most engines update object bbox with root motion/transforms. Update2: actual temporary fix https://github.com/mrdoob/three.js/issues/11991#issuecomment-323610001

Three.js version

• [x] r87
• [x] r85

Browser

• [x] Firefox

OS

• [x] Windows

Hardware Requirements (graphics card, VR Device, ...)

[RemusMar]

I really don't think so. Feel free to check out the skinned meshes and shadows in any position and angle you want: http://necromanthus.com/Test/html5/testA_disco.html

[gogiii]

RemusMar, you're moving your models in code by changing position and rotation which is obvious for games/procedural movement and yes this moves object with its root (which is 0,0,0) when threejs does the frustum culling, that's why it works. But in my case the complete movement is in actual animationclip, there is no movement in code (except starting position I picked randomly to show the bug), the root moves around animation clip itself. It's like your girl walking around as complete baked animation, not a walkcycle animation at (0,0,0) + position/rotation changes in code. You can always get back to my fiddle example which shows the bug.

[WestLangley]

most engines update object bbox with root motion/transforms.

They do? Updates are performed on the GPU, no?

Also, how can updating the bounding box every frame be performant?

[gogiii]

WestLangley, depends on implementation, mostly gpu culling means complex methods including occlusion culling, when you query bounds against current buffer, but this could be too complex to implement in web. But here we have a simple frustum culling, isn't it? As no threejs loaders support more than a single skeleton/armature we don't need to split it between meshes. So the simplest variant is to precompute the biggest bounds from all animation frames on load (this one is a bit rough and not preciese) or to let the user set its own bounds AROUND root bone (simple engines like quake use sphere+aabb checks for frustum). Then the bounds must be transformed by the root bone picked of the current frame - this will be the bounding box to check against the frustum. Actually threejs already does this when it comes to moving the model by position/rotation for simple mesh, same should be also done for root bone when it works with skinned mesh (yes, BOTH position/rotation of Object3D, and skeleton/armature root transform must be taken into calculations) This will mostly be working instead of showing undefined behaviour when you see half of shadow or no model or any combination of these two visible or cut. I wonder how does current frustum culling implementation takes into account model changes in size... it either picks the first frame or reset pose bounds and definetly doesn't transform it about root bone while animating.

[WestLangley]

three.js' implementation of frustum culling is trivial. I suggest you familiarize yourself with it, and then you will be in a position to make a recommendation for improvement.

Your approach of baking the character's position into the animation is an interesting one, but I agree, it is not particularly compatible with the three.js culling approach.

[gogiii]

Your approach of baking the character's position into the animation is an interesting one

Well, baking animations is a standard for in-game cinematics/intros/movies and any non-interactive stuff (ut2003, gow, etc), it's the easiest way to add some movement to your scene.

three.js' implementation of frustum culling is trivial. I suggest you familiarize yourself with it, and then you will be in a position to make a recommendation for improvement.

Well, I guess, all I need is to add root bone transform to the object transform in intersectsObject.

Update: I confirm that the next code will fix both problems. Still this may be conflicting with logics/representation of objects within engine.

intersectsObject: function () {

        var sphere = new Sphere();

        return function intersectsObject( object ) {

            var geometry = object.geometry;

            if ( geometry.boundingSphere === null )
                geometry.computeBoundingSphere();

                sphere.copy( geometry.boundingSphere );

                    if(object.isSkinnedMesh)
                        sphere.applyMatrix4( object.skeleton.bones[0].matrix );

                    sphere.applyMatrix4( object.matrixWorld );

            return this.intersectsSphere( sphere );

        };

    }(),

[RemusMar]

Actually the use case for this was to make a complete animation in any 3d editor (blender in my case) just to play it as is in three.js. So the character isn't moved by mesh.position, there is no need to sync movement/animation and hardcode movement positions. It just plays baked animation!

Update: yes, making frustumCulled = false does fix both problems

Now I see all your (edited) message. Yes, that's the solution for your particular case.

but most engines update object bbox with root motion/transforms.

That's a waste of processing power. Not a good idea for a JS library.

[gogiii]

That's a waste of processing power. RemusMar, not much processing power for a single matrix multiplication.

This will affect the possibility of using any mocap animation (bvh) containing root motion on any models in threejs as it will lead to undefined clipping as most mocap animations are baked and have offsets. Also the closest implementation example of this is 'apply root motion' checkbox in unity3d.

Nevertheless, in current state it's a bug, which may pop up any time in different aspects of engine (currently: skinnedmesh disappearing and shadow clipping in any combination, maybe some reflection techs already suffer).

Probably putting a note in docs that skinnedmesh may be clipped if moved away from its origin by skeleton/armature will help users to see the problem and avoid this. Also a way to redefine own frustum/intersection test by user is one of the simplest solutions instead of making full implementation since I'm not quite sure that my fix will solve all possible use cases.

[Usnul]

There is a bit more to it. If your character's base pose is not T or if your character is not a bi-ped, normal animations will cause clipping. Imagine a horse, when it runs the tail will rise and fall, if you position camera to face horse sideways with tail in fallen pose just outside the frustum - your animation will likely be clipped. This also includes things like running animation for bipeds.

I use method suggested by @gogiii - to precompute bounding volumes based on keyframes. As far as current implementation goes there are 3 ways to combat this bug:

• use infinite bounding volume (frustumCulled = false)
• compute maximum bounding volume ( @gogiii suggested this)
• opportunistic heuristic, some variation of Volume*(1+X) where X is an estimate margin of error.

it's also possible to use a worker thread to compute volume and while computation is running - use infinite volume.

[RemusMar]

use infinite bounding volume (frustumCulled = false)

This is a must for particles anyway. See the fire/smoke and steam implementation in the posted link. If you study the source, you'll find:

        steam.mesh.frustumCulled = false;
        fire.mesh.frustumCulled = false;

In any case, for low-poly meshes, any solution or workaround is an option. But for above 500K triangles ...

[gogiii]

Usnul, the horse case is a bit more close to updating boundings problem than to in-clip movement of character I'm talking about. In my case the boundings not only change in dimensions, but move around animation scene. But... oh, well, I got it. Instead of making changes in frustum.js, alll I need is to update the geometry.boundingSphere on the user side:

function animate() {
    requestAnimationFrame(animate);
    scene.children.forEach((o) => {
        if(o.isSkinnedMesh) {
            o.geometry.computeBoundingSphere();
            var sphere = new THREE.Sphere();
            sphere.copy( o.geometry.boundingSphere );
            sphere.applyMatrix4( o.skeleton.bones[0].matrix );
            o.geometry.boundingSphere = sphere; // easy
        }
    });
    renderer.render(scene, camera);
}

This solves my problem (actually I solved it in the first post by turning off culling at all), but I think I'll leave the issue as probably the engine should be handling both cases somehow on its side.

[chubei]

I run into this problem too. To my knowledge, people in animation industry refer to 'in-clip movement of character' as root motion and some game engines take special care of that when it comes to implementing skeletal animation. For example Unity optionally converts animation of root bone to the animation of mesh, which means the local transform of root bone stays unanimated while local transform of mesh is animated.

We currently work around this by adding a property called 'rootMotionBone' in Skeleton. Anytime we need to use the mesh's bounding box or bounding sphere for culling or picking, we transform its geometry's bounding box or sphere by rootMotionBone's world matrix. Although the geometry's bounding box or sphere may be incorrect because of animation, this method hardly affects performance (only a extra conditioning for checking if the 3DObject has 'rootMotionBone' property) and gives satisfying result most of the time.

[Usnul]

This is a must for particles anyway.

@RemusMar not necessarily. Depending on your motion model, it may be possible to estimate emitter limits. I do this without much difficulty. My implementation is quite simplistic and does not estimate bounds during motion (i.e. using infinite bounds while emitter is moving). Keeping motion of emitter position would require tracking trajectory over longest particle life, which is a bit more complex but is also doable. Particle emitters are pretty expensive, so if you want to be able to use a large number of them - you have to be able to cull efficiently, allowing you to pause simulation and only catch up a portion of paused time as sleepingTime%maxLife when waking up.

[RemusMar]

My implementation is quite simplistic and does not estimate bounds during motion

So it's not accurate. For low-poly models (including particles), "frustumCulled = false" is the best solution by far. Of course, there is always place for improvements.

[donmccurdy]

Should this be considered a duplicate of https://github.com/mrdoob/three.js/issues/14499? Or vice versa? Or is there something additional here?

[Mugen87]

Should this be considered a duplicate of #14499?

I would say yes.

[gogiii]

@donmccurdy this one contains jsfiddle example and two variants of workaround within discussion. @mrdoob current issue also shows that the bug affects both mesh visibility and shadowing.

[mrdoob]

@gogiii thanks for creating the jsfiddle 🙏