search

vrm-c / vrm-specification

vrm specification
235 stars 37 forks source link

[FAQ] How to read bone orientation and position? #371

Closed vivi90 closed 2 years ago

vivi90 commented 2 years ago

How to read the positions and rotations of each bone?

Because every model might use different different body proportions, i need the read the position and rotation of every bone from the *.vrm (binary glTF) model file.

I failed to do that. It seems there is something missing in the documentation or i understand it the wrong way.

vivi90 commented 2 years ago

What i tried:

  1. Rename *.vrm to *.glb.
  2. Convert *.glb to *.gltf (JSON).
  3. Then i get this one (shortened): 
    {
"extensionsUsed": ["KHR_materials_unlit", "VRM"],
"accessors": [..],
"asset": {..},
"buffers": [..],
"materials": [..],
"nodes": [
{
  "mesh": 0,
  "name": "Face"
}, 
{
  "mesh": 1,
  "name": "Body"
}, 
{
  "mesh": 2,
  "name": "Hair"
}
],
"samplers": [..],
"scene": 0,
"scenes": [
{
  "nodes": [0, 1, 2]
}
],
"textures": [..],
"extensions": {
"VRM": {
  "exporterVersion": "VRoid Studio-1.7.0",
  "specVersion": "0.0",
  "meta": {..},
  "humanoid": {
    "humanBones": [
      {
        "bone": "hips",
        "node": 1,
        "useDefaultValues": true
      }, 
      {
        "bone": "leftUpperLeg",
        "node": 101,
        "useDefaultValues": true
      },
      ..
    ],
    "armStretch": 0.05,
    "legStretch": 0.05,
    "upperArmTwist": 0.5,
    "lowerArmTwist": 0.5,
    "upperLegTwist": 0.5,
    "lowerLegTwist": 0.5,
    "feetSpacing": 0,
    "hasTranslationDoF": false
  },
  "firstPerson": {..},
  "blendShapeMaster": {..},
  "secondaryAnimation": {
    "boneGroups": [..],
    "colliderGroups": [
      {
        "node": 2,
        "colliders": [
          {
            "offset": {
              "x": 0,
              "y": 0,
              "z": 0
            },
            "radius": 0.114335559
          }
        ]
      }, 
      {
        "node": 4,
        "colliders": [
          {
            "offset": {
              "x": 0,
              "y": 0.003161192,
              "z": 0.008988265
            },
            "radius": 0.095279634
          }, 
          {
            "offset": {
              "x": -0.047639817,
              "y": 0.0552625656,
              "z": -0.0295360088
            },
            "radius": 0.06669574
          }, 
          {
            "offset": {
              "x": 0.047639817,
              "y": 0.0552625656,
              "z": -0.0295360088
            },
            "radius": 0.06669574
          }
        ]
      },
      ..
    ]
  },
  "materialProperties": [..]
}
}
}

According to VRM 0.0 and VRM 2.0 beta now i should do the following steps:

  1. read extensions.VRM.humanoid.humanBones[*].node
  2. Using the return values for nodes[*] and get according to VRM 0.0 specification for each something like: 
    {
"name": "..",
"position": [.., .., ..],
"rotation": [.., .., .., ..],
"scale": [.., .., ..]
}

    Thats exactly what i want!

But what i get for each node is this (please see the code above for more details):

{
  "mesh": ..,
  "name": ".."
}

Also only for the Hair, Body and Face bone groups (instead of each bone) and without rotation and position at all. 🙁

So, what's wrong here? Please give me some instructions. 🙂

mrxz commented 2 years ago

According to the spec of VRM 0.0, the model must be in a T-pose and bones must not have any scale or (local) rotation (spec). So it isn't strange that these properties might be omitted (they'd be [0, 0, 0, 1] and [1, 1, 1] anyway)

If there is no translation property (spec incorrectly calls it position), that would mean the position is [0, 0, 0]. This can be the case for several nodes, but you would expect at least some translations on some of the nodes.

Also only for the Hair, Body and Face bone groups (instead of each bone) and without rotation and position at all.

Do you only have these three nodes? If so, something must've gone wrong when you converted the .glb file into .gltf or the .vrm file was corrupt/invalid to begin with.

In any case, it's worth pointing out that the translation of a node is only the local translation. So to know the position of a bone(/node) in model space, you'd have to add up all the translations of the parent nodes as well. Normally you'd have to take rotation and scaling into account as well, but these should all be identity, so a simple sum of translations should suffice.

vivi90 commented 2 years ago

@mrxz

According to the spec of VRM 0.0, the model must be in a T-pose and bones must not have any scale or (local) rotation (spec). So it isn't strange that these properties might be omitted (they'd be [0, 0, 0, 1] and [1, 1, 1] anyway)

Ah, i see.

If there is no translation property (spec incorrectly calls it position) [..]

Oh okay, thank you.

Also only for the Hair, Body and Face bone groups (instead of each bone) and without rotation and position at all.

Do you only have these three nodes? If so, something must've gone wrong when you converted the .glb file into .gltf or the .vrm file was corrupt/invalid to begin with.

Oh yes, the conversion result was broken. I used this page for quick conversion (very bad idea!): https://products.aspose.app/3d/de/conversion/glb-to-gltf

Now i have done it with Python (requires gltflib) this way:

#!/usr/bin/env python3

from gltflib import GLTF

gltf = GLTF.load_glb(
    "test.vrm",
    load_file_resources = True
)
gltf.convert_to_file_resource(
    gltf.get_glb_resource(), 
    "resources.bin"
)
gltf.export(
    "test.gltf", 
    save_file_resources=True
)

And get this one:

{
  "extensions": {
    "VRM": {
      ..
      "humanoid": {
        "humanBones": [
          {
            "bone": "hips",
            "node": 1,
            "useDefaultValues": true
          },
          {
            "bone": "leftUpperLeg",
            "node": 101,
            "useDefaultValues": true
          },
          ..
        ],
        ..
      },
      ..
    }
  },
  ..
  "nodes": [
    {
      "name": "Root",
      "children": [1],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [0, 0, 0]
    },
    {
      "name": "J_Bip_C_Hips",
      "children": [2, 101, 125],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [0, 0.907995462, -0.00357606518]
    },
    ..
    {
      "name": "J_Bip_L_UpperLeg",
      "children": [102, 104, 106, 108],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [-0.07715603, -0.03977418, 0.00365623739]
    },
    ..
  ],
  ..
}

Yeah, nodes available now.

One problem still exists: The relation between the humanBones is defined by "children": [..] in nodes[*]. But how to query the corresponding node of each humanBone?

In any case, it's worth pointing out that the translation of a node is only the local translation. So to know the position of a bone(/node) in model space, you'd have to add up all the translations of the parent nodes as well. Normally you'd have to take rotation and scaling into account as well, but these should all be identity, so a simple sum of translations should suffice.

Good to know 🙂

0b5vr commented 2 years ago

The relation between the humanBones is defined by "children": [..] in nodes[*]. But how to query the corresponding node of each humanBone?

I'm afraid I don't understand your intention. what /nodes/i/children defines is a relationship between nodes of glTF, not humanBones in the VRM extension. And /extensions/VRM/humanoid/humanBones/i/node represents the index of the corresponding glTF node, as you have mentioned.

vivi90 commented 2 years ago

@0b5vr

The relation between the humanBones is defined by "children": [..] in nodes[*]. But how to query the corresponding node of each humanBone?

I'm afraid I don't understand your intention. what /nodes/i/children defines is a relationship between nodes of glTF, not humanBones in the VRM extension.

Yes, thats clear.

And /extensions/VRM/humanoid/humanBones/i/node represents the index of the corresponding glTF node, as you have mentioned.

No, i don't see it in my glTF:

{
  "extensions": {
    "VRM": {
      ..
      "humanoid": {
        "humanBones": [
          {
            "bone": "hips",
            "node": 1,
            "useDefaultValues": true
          },
          {
            "bone": "leftUpperLeg",
            "node": 101,
            "useDefaultValues": true
          },
          ..
        ],
        ..
      },
      ..
    }
  },
  ..
  "nodes": [
    {
      "name": "Root",
      "children": [1],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [0, 0, 0]
    },
    {
      "name": "J_Bip_C_Hips",
      "children": [2, 101, 125],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [0, 0.907995462, -0.00357606518]
    },
    ..
    {
      "name": "J_Bip_L_UpperLeg",
      "children": [102, 104, 106, 108],
      "rotation": [0, 0, 0, 1],
      "scale": [1, 1, 1],
      "translation": [-0.07715603, -0.03977418, 0.00365623739]
    },
    ..
  ],
  ..
}

Nothing there connects the VRM bone with it's node. Thats my problem 🙁

0b5vr commented 2 years ago

this part:

        "humanBones": [
          {
            "bone": "hips",
            "node": 1,
            "useDefaultValues": true
          },
          {
            "bone": "leftUpperLeg",
            "node": 101,
            "useDefaultValues": true
          },
          ...
        ]

"node": 1 must represent that /nodes/1 is hips, and "node": 101 represents that /nodes/101 is the left upper leg.

I guess the node J_Bip_L_UpperLeg is 101st (or 102nd if you count from one) node of the /nodes .

Note that /nodes/i side does not have any information that ties the node itself to any VRM humanoid bones.

vivi90 commented 2 years ago

"node": 1 must represent that /nodes/1 is hips, and "node": 101 represents that /nodes/101 is the left upper leg.

gltf.model.nodes[101]

Oh yes, it's in the right order 😆

I guess the node J_Bip_L_UpperLeg is 101st (or 102nd if you count from one) node of the /nodes .

It's 101 because the root node is 0.

Note that /nodes/i side does not have any information that ties the node itself to any VRM humanoid bones.

Yes, thats right.

Okay, seems solved for me, thank you! 🥳