SkinnedMesh support

[enyo]

Hi!

I tried using a gltf with a skinned mesh, and the result looks like this:

These are the correct normals:

And here, the surface id debug buffer:

So as you can see, the skinned mesh simply doesn’t get a surface id.

Is there an easy way to get this to work?

[OmarShehata]

Thanks for reporting this @enyo ! Are you able to share this model, or a similar model that has the same issue for testing?

This is where the code is for iterating over all vertices to compute the surface IDs. It's possible it isn't taking into account this piece of the mesh? It must be stored differently in glTF/in how ThreeJS loads it?

https://github.com/OmarShehata/webgl-outlines/blob/main/threejs-outlines-minimal/src/FindSurfaces.js#L22

[LancerComet]

For SkinnedMesh there is no easy way to make it work because you have to deal with the bone manually (skinning). It is a little bit complex but it can still be done anyway. You have to modify the SurfaceFinder to apply skinning.

I have a project that already supports playing AnimationClips. The prerequisite for it is to properly handle the SkinnedMesh. The code for handling SkinnedMesh is quite extensive, but here is the core logic of the skinning process:

private static _applySkinning (
    skinnedMesh: THREE.SkinnedMesh,
    positionAttribute: THREE.BufferAttribute | THREE.InterleavedBufferAttribute
  ): Float32Array {
    const boneMatrices = skinnedMesh.skeleton.boneMatrices
    const bindMatrix = skinnedMesh.bindMatrix
    const bindMatrixInverse = skinnedMesh.bindMatrixInverse

    const vertexCount = positionAttribute.count
    const transformedPositions = new Float32Array(vertexCount * 3)

    const tempVertex = new THREE.Vector3()
    const skinnedVertex = new THREE.Vector3()
    const tempMatrix = new THREE.Matrix4()
    const skinIndex = new THREE.Vector4()
    const skinWeight = new THREE.Vector4()

    for (let i = 0; i < vertexCount; i++) {
      tempVertex.fromBufferAttribute(positionAttribute, i)
      tempVertex.applyMatrix4(bindMatrix)

      // TODO: Might crash.
      skinIndex.fromBufferAttribute(skinnedMesh.geometry.attributes.skinIndex as THREE.BufferAttribute, i)
      skinWeight.fromBufferAttribute(skinnedMesh.geometry.attributes.skinWeight as THREE.BufferAttribute, i)

      skinnedVertex.set(0, 0, 0)

      for (let j = 0; j < 4; j++) {
        const weight = skinWeight.getComponent(j)
        if (weight !== 0) {
          const boneIndex = skinIndex.getComponent(j)
          const offset = boneIndex * 16

          for (let k = 0; k < 16; k++) {
            tempMatrix.elements[k] = boneMatrices[offset + k]
          }

          const transformedVertex = tempVertex.clone().applyMatrix4(tempMatrix).multiplyScalar(weight)
          skinnedVertex.add(transformedVertex)
        }
      }

      skinnedVertex.applyMatrix4(bindMatrixInverse)
      transformedPositions.set([skinnedVertex.x, skinnedVertex.y, skinnedVertex.z], i * 3)
    }

    return transformedPositions
  }

This is what it would look like once you get it working:

https://github.com/OmarShehata/webgl-outlines/assets/10321350/73a6a022-4adc-4ed3-b603-65ecbf4d2ba3