[FR] VMC protocol support

[vivi90]

Feature request: VMC protocol export support https://protocol.vmc.info/english

Ready to help, implementing it.

[Arthur151]

I don't know what is VMC developed for ? But it seems interesting. What do we need to achieve this?

[vivi90]

@Arthur151 Sorry for my late answer.

I don't know what is VMC developed for ?

To keep it short: The VMC protocol is based on the OSC protocol. Established with the goal to create an realtime connection standard between different motion capture and 3D avatar rendering solutions (beside other features). It's mainly used by vTubers.

What do we need to achieve this?

In short: Not much. At least required are the skeleton bone positions according to the VRM specs.

Let's do it this way: At first I will provide an python package for handling all required basic VMC communication. After that, i answer you again with more concrete info about how to use this package in ROMP. What do you think?

[Arthur151]

It seems interesting. Looking forward to your detailed introduction.

[vivi90]

@Arthur151 Hallo my friend 🙂

[..] Looking forward to your detailed introduction.

We have two possibilities to implement the VMC protocol.

a) Sending all skeleton bones with it's Position(x, y, z) and Quaternion(x, y, z, w)

Valid bones

• Hips
• LeftUpperLeg
• RightUpperLeg
• LeftLowerLeg
• RightLowerLeg
• LeftFoot
• RightFoot
• SpineRightFoot
• Chest
• Neck
• Head
• LeftShoulder
• RightShoulder
• LeftUpperArm
• RightUpperArm
• LeftLowerArm
• RightLowerArm
• LeftHand
• RightHand
• LeftToes
• RightToes
• LeftEye
• RightEye
• Jaw
• LeftThumbProximal
• LeftThumbIntermediate
• LeftThumbDistal
• LeftIndexProximal
• LeftIndexIntermediate
• LeftIndexDistal
• LeftMiddleProximal
• LeftMiddleIntermediate
• LeftMiddleDistal
• LeftRingProximal
• LeftRingIntermediate
• LeftRingDistal
• LeftLittleProximal
• LeftLittleIntermediate
• LeftLittleDistal
• RightThumbProximal
• RightThumbIntermediate
• RightThumbDistal
• RightIndexProximal
• RightIndexIntermediate
• RightIndexDistal
• RightMiddleProximal
• RightMiddleIntermediate
• RightMiddleDistal
• RightRingProximal
• RightRingIntermediate
• RightRingDistal
• RightLittleProximal
• RightLittleIntermediate
• RightLittleDistal

• UpperChest

  Pros Exact pose.

  Cons The by ROMP estimated bones needs to match the bones of the used VRM model. The local positions of these model bones depends on the used model by the user. The good news: We are able to read these values from the users model file (https://github.com/vrm-c/vrm-specification/issues/371).

  b) Sending only some keypoints as virtual trackers with it's Position(x, y, z) and Quaternion(x, y, z, w).

  Valid trackers

• Head
• Spine
• Pelvis
• LeftHand
• RightHand
• LeftFoot

• RightFoot

  Pros Don't need to match the model bones.

  Cons No exact pose.

I have taken an quick look into the ROMP FBX export script. But i don't know how diffcult it is to implement a) or b). Solution a) implies maybe some sort of IK (inverse kinematics), if not yet implemented. On the other hand might solution b) be good enough.

So what do you think? How about testing both solutions?

The communication protocol itself is easy.

[Arthur151]

Hi, Vivian,

Glad to hear from you!

Yes, you can get all keypoints' 3D location and their quaternion from this function https://github.com/Arthur151/ROMP/blob/c8c83343265201b46f4e3ed31b684ee3b4db3c47/simple_romp/tools/convert2fbx.py#L155

I think the challenge is about getting all joints we need from estimated SMPL mesh. I think the default skeleton of each character might be different from corresponding joints of SMPL. So it would be wonderful if we have a skeleton of VMC rigged to SMPL model. This could be the perfect solution. Some automatric rigging algorithm/software(Blender) could do this.

[vivi90]

@Arthur151

Yes, you can get all keypoints' 3D location and their quaternion from this function

https://github.com/Arthur151/ROMP/blob/c8c83343265201b46f4e3ed31b684ee3b4db3c47/simple_romp/tools/convert2fbx.py#L155

How to use it?

I think the challenge is about getting all joints we need from estimated SMPL mesh. I think the default skeleton of each character might be different from corresponding joints of SMPL.

i think the same way.

So it would be wonderful if we have a skeleton of VMC rigged to SMPL model. This could be the perfect solution. Some automatric rigging algorithm/software(Blender) could do this.

Unfortunately i am absolutely not familiar with rigging, SMPL or it's mathematics. Have already taken a look at the SMPL docu but don't unterstand it.

[vivi90]

[..] a skeleton of VMC rigged to SMPL model [..]

You mean VRM. VRM is the 3D Avatar model (extended binary glTF). VMC is the transmission protocol (extended OSC) for position and quaterion transforms.

[Arthur151]

For the first step, I think you can just take the video predictions from simple-romp for animation. You can take the 3D keypoints ('joints') and convert 'smpl_thetas' to quaterion.

[vivi90]

For the first step, I think you can just take the video predictions from simple-romp for animation. You can take the 3D keypoints ('joints') and convert 'smpl_thetas' to quaterion.

Thank you but i don't understand how your code for SMPL works. So i am not able to implement it by myself. Maybe some more comments in the code may help. 🙂

[vivi90]

So it would be wonderful if we have a skeleton of VMC rigged to SMPL model. This could be the perfect solution. Some automatric rigging algorithm/software(Blender) could do this.

Unfortunately the VMC protocol specification is fairly unclear about this and calibration. I asked it's author and he pointed me to the reference implementation: https://github.com/sh-akira/VirtualMotionCapture/blob/master/Assets/Scripts/ExternalSender/ExternalSender.cs

So for me there is no information at all about it.

[Arthur151]

Then we can start with solution 2 then.

1. Firstly please run simple-romp on demo videos to get motion sequence saved in video_results.npz.
2. Please refer to https://github.com/Arthur151/ROMP/blob/c8c83343265201b46f4e3ed31b684ee3b4db3c47/simple_romp/tools/convert2fbx.py#L155 to convert the results to the format you want.

[vivi90]

Then we can start with solution 2 then.

Okay.

1. Firstly please run simple-romp on demo videos to get motion sequence saved in video_results.npz.

Using the realtime webcam tracking would be nice 🙂

2. Please refer to https://github.com/Arthur151/ROMP/blob/c8c83343265201b46f4e3ed31b684ee3b4db3c47/simple_romp/tools/convert2fbx.py#L155

   to convert the results to the format you want.

I need the bone name, position (x, y, z) and the quaternion (x, y, z, w). Euler rotation angles are also okay, then i only need to calculate the quaternion.

[Arthur151]

Sure, you can firstly using video API for testing.

Bone name https://github.com/Arthur151/ROMP/tree/master/simple_romp#joints-in-output-npz-file

position is the key joints in outputs / saved .npz

The vector-angle 3D rotation of 24 SMPL joints are smpl_thetas You can convert to any format you like via https://github.com/Arthur151/ROMP/blob/60782b20c8cf80daab75c111f4b7e6e8414a1400/simple_romp/romp/utils.py#L682

That's it, all the things you need. My job is done. Let's see your part.

[vivi90]

@Arthur151 After taking a look at this line: https://github.com/Arthur151/ROMP/blob/c8c83343265201b46f4e3ed31b684ee3b4db3c47/simple_romp/tools/convert2fbx.py#L353 I had an spontaneous idea (not tested): The VMC protocol is mainly used with *.fbx and *.vrm (binary glTF) models. For example by using the following snippet, it's easy to extract bones and it's positions of the user's VRM model:

#!/usr/bin/env python3

from gltflib import GLTF

gltf = GLTF.load_glb(
    "test.vrm",
    load_file_resources = False
)
print(gltf.model.nodes[101]) # leftUpperLeg

(more details: https://github.com/vrm-c/vrm-specification/issues/371)

So how about to:

1. add an option to load the users model at the beginning (by using bpy.ops.import_scene.gltf()).
2. combining the extracted bone positions from the user's model with the estimated bone quaternions.
3. just sending the results 'as it is' via VMC protocol.

I will test it soon, just wanted to share my thoughts. 🙂

[vivi90]

Oh important: It seems i missed to mention, that the VMC protocol only supports single-person applications. Note for users: This requires to use Simple ROMP with the --show_largest option: https://github.com/Arthur151/ROMP/blob/e937a81fa68ccd0fe4adde5c39bc9f432dda8c98/simple_romp/romp/main.py#L27

[Arthur151]

Hi @vivi90 , Happy to hear from you~ Thanks for your efforts on developing such a great extension. Looking forward to see it. Best, Yu

[vivi90]

@Arthur151 I have taken an closer look at the rigs (from left: VRM, SMPL v1.0.0, Mixamo): It seems you are ignoring the ..root joint (f_avg_root or m_avg_root) from the SMPL v1.0.0 FBX model (for Unity)? https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L67 https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L172 https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L304 And then looking for it by the wrong name: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L308 Is this a bug? 🙂

[vivi90]

@Arthur151 It seems, we did not require Blender or bpy just for the VMC protocol. If i understand convert2fbx.py the right way, then it doesn't do any vertex estimation from the *.npz file. Only bone rotations and the root bone translation. If thats the point, then everything that is there done by bpy can also achieved by gltflib (of course not for *.fbx).

Note: I see in np.load(input_path, allow_pickle=True)['results'][()][frame_name]['smpl_thetas'][0] the following:

{
  'smpl_thetas': array(
    [
      [
        2.99169707e+00, -1.14872180e-01, 2.52191164e-02,
        -5.85280538e-01, -2.17623878e-02, 1.04836568e-01,
        -5.94840944e-01, 2.05031112e-02, -4.04515043e-02,
        4.90758419e-01, -1.56615092e-03, -1.22337053e-02,
        1.18982077e+00, 6.25830796e-03, -1.33860081e-01,
        1.03421855e+00, -4.33490239e-02, 9.75086764e-02,
        8.08686297e-03, -1.64354779e-02, -1.56924091e-02,
        -5.29559441e-02, 1.50208578e-01, -8.31045508e-02,
        -6.23615980e-02, -1.55124173e-01, 8.85685012e-02,
        3.80534641e-02, -1.49868811e-02, -1.34220216e-02,
        -2.74092108e-01, 9.23865065e-02, 1.74044952e-01,
        -2.23217666e-01, -2.57145371e-02, -1.95539817e-01,
        -1.50648043e-01, 2.01505367e-02, -2.51643322e-02,
        3.70577164e-02, -2.34452099e-01, -3.42204958e-01,
        2.55778432e-02, 2.65333027e-01, 2.82382011e-01,
        8.90259165e-03, 5.71487285e-03, 1.26959458e-02,
        9.14356261e-02, -3.04939717e-01, -9.25578892e-01,
        1.75735921e-01, 3.51879328e-01, 9.20472264e-01,
        6.75733909e-02, -5.05667686e-01, 6.31040707e-02,
        2.48549446e-01, 8.01332653e-01, -9.67753083e-02,
        7.34215826e-02, -6.02314919e-02, 1.28485382e-01,
        5.53916991e-02, 8.01604688e-02, -1.09875299e-01,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00,
        0.00000000e+00, 0.00000000e+00, 0.00000000e+00
      ]
    ], 
    dtype = float32
  )
}

It's not Quaternions or Euler degrees. What exactly is it? I see it's processed by the following lines but still unsure: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L157 What means 72? https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L162 Okay, after that it's an rotation matrix. And here: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L183 It becomes finally an Quaternion.

[vivi90]

@Arthur151 I need a little bit help. I wanted to use *.npz files, created by an video (see: https://github.com/Arthur151/ROMP/issues/267):

romp -t -sc=3 --onnx --show_largest --mode=video -i "E:\Projekte\romp-test\test.mp4" --calc_smpl --render_mesh --save_video -o "E:\Projekte\romp-test\test_result.mp4"

According to: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L211 and: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L212

I get in the variable frame_results for the frame files (for example 00000215.npz) valid results and sequence_results is invalid (KeyError: 'sequence_results is not a file in the archive'). Seems okay to me.

But if i am testing it with the final *.npz file (for example video_results.npz), then i get for both (frame_results and sequence_results) invalid results: _pickle.UnpicklingError: Failed to interpret file 'video_results.npz' as a pickle

What's my mistake? 🙂

[vivi90]

@Arthur151 Sorry, my mistake. Seems to load properly, if i edit the script a little bit and open it inside Blender:

# -*- coding: utf-8 -*-

# Max-Planck-Gesellschaft zur Förderung der Wissenschaften e.V. (MPG) is
# holder of all proprietary rights on this computer program.
# You can only use this computer program if you have closed
# a license agreement with MPG or you get the right to use the computer
# program from someone who is authorized to grant you that right.
# Any use of the computer program without a valid license is prohibited and
# liable to prosecution.
#
# Copyright©2019 Max-Planck-Gesellschaft zur Förderung
# der Wissenschaften e.V. (MPG). acting on behalf of its Max Planck Institute
# for Intelligent Systems. All rights reserved.
#
# Contact: ps-license@tuebingen.mpg.de
#
# Author: Joachim Tesch, Max Planck Institute for Intelligent Systems, Perceiving Systems
#
# Create keyframed animated skinned SMPL mesh from .pkl pose description
#
# Generated mesh will be exported in FBX or glTF format
#
# Notes:
#  + Male and female gender models only
#  + Script can be run from command line or in Blender Editor (Text Editor>Run Script)
#  + Command line: Install mathutils module in your bpy virtualenv with 'pip install mathutils==2.81.2'

import os
import sys
import time
import argparse
import numpy as np
from math import radians

try:
    import bpy
except:
    print('Missing bpy, install via pip, please install bpy by yourself if failed.')
    os.system('pip install future-fstrings')
    os.system('pip install tools/bpy-2.82.1 && post_install')
    import bpy
try:
    from mathutils import Matrix, Vector, Quaternion, Euler
except:
    os.system('pip install mathutils==2.81.2')
    from mathutils import Matrix, Vector, Quaternion, Euler

# Globals
# Add your UNIX paths here!
male_model_path = '/home/yusun/ROMP/model_data/SMPL_unity_v.1.0.0/smpl/Models/SMPL_m_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx'
female_model_path = 'E:/Projekte/romp-test3/SMPL_f_unityDoubleBlends_lbs_10_scale5_207_v1.0.0.fbx'
character_model_path = None

'''
python tools/convert2fbx.py --input=/home/yusun/BEV_results/video_results.npz --output=/home/yusun/BEV_results/dance.fbx --gender=female
'''

fps_source = 24
fps_target = 24

gender = 'female' #female

support_formats = ['.fbx', '.glb', '.bvh']

vrm_bone_names = (
    0 : "Hips",
    1 : "LeftUpperLeg",
    2 : "RightUpperLeg",
    3 : "Spine",
    4 : "LeftLowerLeg",
    5 : "RightLowerLeg",
    6 : "Chest",
    7 : "LeftFoot",
    8 : "RightFoot",
    9 : "UpperChest"
    10 : "LeftToes",
    11 : "RightToes",
    12 : "Neck",
    13 : "LeftIndexProximal",
    14 : "RightIndexProximal",
    15 : "Head",
    16 : "LeftShoulder",
    17 : "RightShoulder",
    18 : "LeftUpperArm",
    19 : "RightUpperArm",
    20 : "LeftLowerArm",
    21 : "RightLowerArm",
    22 : "LeftHand",
    23 : "RightHand"
)

bone_name_from_index = {
    0 : 'Pelvis',
    1 : 'L_Hip',
    2 : 'R_Hip',
    3 : 'Spine1',
    4 : 'L_Knee',
    5 : 'R_Knee',
    6 : 'Spine2',
    7 : 'L_Ankle',
    8: 'R_Ankle',
    9: 'Spine3',
    10: 'L_Foot',
    11: 'R_Foot',
    12: 'Neck',
    13: 'L_Collar',
    14: 'R_Collar',
    15: 'Head',
    16: 'L_Shoulder',
    17: 'R_Shoulder',
    18: 'L_Elbow',
    19: 'R_Elbow',
    20: 'L_Wrist',
    21: 'R_Wrist',
    22: 'L_Hand',
    23: 'R_Hand'
}

# To use other avatar for animation, please define the corresponding 3D skeleton like this.
bone_name_from_index_character = {
    0 : 'Hips',
    1 : 'RightUpLeg',
    2 : 'LeftUpLeg',
    3 : 'Spine',
    4 : 'RightLeg',
    5 : 'LeftLeg',
    6 : 'Spine1',
    7 : 'RightFoot',
    8: 'LeftFoot',
    9: 'Spine2',
    10: 'LeftToeBase',
    11: 'RightToeBase',
    12: 'Neck',
    13: 'LeftHandIndex1',
    14: 'RightHandIndex1',
    15: 'Head',
    16: 'LeftShoulder',
    17: 'RightShoulder',
    18: 'LeftArm',
    19: 'RightArm',
    20: 'LeftForeArm',
    21: 'RightForeArm',
    22: 'LeftHand',
    23: 'RightHand'
}

# Helper functions

# Computes rotation matrix through Rodrigues formula as in cv2.Rodrigues
# Source: smpl/plugins/blender/corrective_bpy_sh.py
def Rodrigues(rotvec):
    theta = np.linalg.norm(rotvec)
    r = (rotvec/theta).reshape(3, 1) if theta > 0. else rotvec
    cost = np.cos(theta)
    mat = np.asarray([[0, -r[2], r[1]],
                      [r[2], 0, -r[0]],
                      [-r[1], r[0], 0]])
    return(cost*np.eye(3) + (1-cost)*r.dot(r.T) + np.sin(theta)*mat)

# Setup scene
def setup_scene(model_path, fps_target):
    scene = bpy.data.scenes['Scene']

    ###########################
    # Engine independent setup
    ###########################

    scene.render.fps = fps_target

    # Remove default cube
    if 'Cube' in bpy.data.objects:
        bpy.data.objects['Cube'].select_set(True)
        bpy.ops.object.delete()

    # Import gender specific .fbx template file
    bpy.ops.import_scene.fbx(filepath=model_path)

# Process single pose into keyframed bone orientations
def process_pose(current_frame, pose, trans, pelvis_position):

    if pose.shape[0] == 72:
        rod_rots = pose.reshape(24, 3)
    else:
        rod_rots = pose.reshape(26, 3)

    mat_rots = [Rodrigues(rod_rot) for rod_rot in rod_rots]

    # Set the location of the Pelvis bone to the translation parameter
    armature = bpy.data.objects['Armature']
    bones = armature.pose.bones

    # Pelvis: X-Right, Y-Up, Z-Forward (Blender -Y)
    root_location = Vector(
        (100*trans[1], 100*trans[2], 100*trans[0])) - pelvis_position
    # Set absolute pelvis location relative to Pelvis bone head
    bones[bone_name_from_index[0]].location = root_location

    # bones['Root'].location = Vector(trans)
    bones[bone_name_from_index[0]].keyframe_insert('location', frame=current_frame)

    for index, mat_rot in enumerate(mat_rots, 0):
        if index >= 24:
            continue

        bone = bones[bone_name_from_index[index]]

        bone_rotation = Matrix(mat_rot).to_quaternion()
        quat_x_90_cw = Quaternion((1.0, 0.0, 0.0), radians(-90))
        #quat_x_n135_cw = Quaternion((1.0, 0.0, 0.0), radians(-135))
        #quat_x_p45_cw = Quaternion((1.0, 0.0, 0.0), radians(45))
        #quat_y_90_cw = Quaternion((0.0, 1.0, 0.0), radians(-90))
        quat_z_90_cw = Quaternion((0.0, 0.0, 1.0), radians(-90))

        if index == 0:
            # Rotate pelvis so that avatar stands upright and looks along negative Y avis
            bone.rotation_quaternion = (quat_x_90_cw @ quat_z_90_cw) @ bone_rotation
        else:
           bone.rotation_quaternion = bone_rotation

        bone.keyframe_insert('rotation_quaternion', frame=current_frame)

    return

# Process all the poses from the pose file
def process_poses(
        input_path,
        gender,
        fps_source,
        fps_target,
        subject_id=-1):

    print('Processing: ' + input_path)

    frame_results = np.load(input_path, allow_pickle=True)['results'][()]
    sequence_results = np.load(input_path, allow_pickle=True)['sequence_results'][()] 

    poses, trans = [], []

    if len(sequence_results)>0:
        subject_ids = list(sequence_results.keys())
        if subject_id == -1 or subject_id not in subject_ids:
            print('Get motion sequence with subject IDs:', subject_ids)
            subject_id = int(input('Please select one subject ID (int):'))
        poses = np.array(sequence_results[subject_id]['smpl_thetas'])
        trans = np.array(sequence_results[subject_id]['cam_trans'])
    else:
        print('Missing tracking IDs in results. Using the first pose results for animation.')
        print('To get the tracking IDs, please use temporal optimization during inference.')
        frame_names = sorted(list(frame_results.keys()))
        poses, trans = np.zeros((len(frame_names), 72)), np.zeros((len(frame_names), 3))
        for inds, frame_name in enumerate(frame_names):
            poses[inds] = frame_results[frame_name]['smpl_thetas'][0]
            trans[inds] = frame_results[frame_name]['cam_trans'][0]

    if gender == 'female':
        model_path = female_model_path
        for k,v in bone_name_from_index.items():
            bone_name_from_index[k] = 'f_avg_' + v
    elif gender == 'male':
        model_path = male_model_path
        for k,v in bone_name_from_index.items():
            bone_name_from_index[k] = 'm_avg_' + v
    elif gender == 'character':
        model_path = character_model_path
        for k, v in bone_name_from_index_character.items():
            bone_name_from_index[k] = 'mixamorig1:' + v
    else:
        print('ERROR: Unsupported gender: ' + gender)
        sys.exit(1)

    # Limit target fps to source fps
    if fps_target > fps_source:
        fps_target = fps_source

    print('Gender:',gender)
    print('Number of source poses: ',poses.shape[0])
    print('Source frames-per-second: ', fps_source)
    print('Target frames-per-second: ', fps_target)
    print('--------------------------------------------------')

    setup_scene(model_path, fps_target)

    scene = bpy.data.scenes['Scene']
    sample_rate = int(fps_source/fps_target)
    scene.frame_end = (int)(poses.shape[0]/sample_rate)

    # Retrieve pelvis world position.
    # Unit is [cm] due to Armature scaling.
    # Need to make copy since reference will change when bone location is modified.
    armaturee = bpy.data.armatures[0]
    ob = bpy.data.objects['Armature']
    armature = ob.data

    bpy.ops.object.mode_set(mode='EDIT')
    # get specific bone name 'Bone'
    pelvis_bone = armature.edit_bones[bone_name_from_index[0]]
    # pelvis_bone = armature.edit_bones['f_avg_Pelvis']
    pelvis_position = Vector(pelvis_bone.head)
    bpy.ops.object.mode_set(mode='OBJECT')

    source_index = 0
    frame = 1

    offset = np.array([0.0, 0.0, 0.0])

    while source_index < poses.shape[0]:
        print('Adding pose: ' + str(source_index))

        # Go to new frame
        scene.frame_set(frame)

        process_pose(frame, poses[source_index], (trans[source_index] - offset), pelvis_position)
        source_index += sample_rate
        frame += 1

    return frame

def rotate_armature(use):
    if use == True:
        # Switch to Pose Mode
        bpy.ops.object.posemode_toggle()

        # Find the Armature & Bones
        ob = bpy.data.objects['Armature']
        armature = ob.data
        bones = armature.bones
        rootbone = bones[0]

        # Find the Root bone
        for bone in bones:
            if "f_avg_root" in bone.name:
                rootbone = bone

        rootbone.select = True

        # Rotate the Root bone by 90 euler degrees on the Y axis. Set --rotate_Y=False if the rotation is not needed.
        bpy.ops.transform.rotate(value=1.5708, orient_axis='Y', orient_type='GLOBAL', orient_matrix=((1, 0, 0), (0, 1, 0), (0, 0, 1)), orient_matrix_type='GLOBAL', constraint_axis=(False, True, False), mirror=True, use_proportional_edit=False, proportional_edit_falloff='SMOOTH', proportional_size=1, use_proportional_connected=False, use_proportional_projected=False, release_confirm=True)
        # Revert back to Object Mode
        bpy.ops.object.posemode_toggle()

def export_animated_mesh(output_path, rotate_y):
    # Create output directory if needed
    output_dir = os.path.dirname(output_path)
    if not os.path.isdir(output_dir):
        os.makedirs(output_dir, exist_ok=True)

    # Fix Rotation
    rotate_armature(rotate_y)

    # Select only skinned mesh and rig
    bpy.ops.object.select_all(action='DESELECT')
    bpy.data.objects['Armature'].select_set(True)
    bpy.data.objects['Armature'].children[0].select_set(True)

    if output_path.endswith('.glb'):
        print('Exporting to glTF binary (.glb)')
        # Currently exporting without shape/pose shapes for smaller file sizes
        bpy.ops.export_scene.gltf(filepath=output_path, export_format='GLB', export_selected=True, export_morph=False)
    elif output_path.endswith('.fbx'):
        print('Exporting to FBX binary (.fbx)')
        bpy.ops.export_scene.fbx(filepath=output_path, use_selection=True, add_leaf_bones=False)
    elif output_path.endswith('.bvh'):
        bpy.ops.export_anim.bvh(filepath=output_path, root_transform_only=False)
    else:
        print('ERROR: Unsupported export format: ' + output_path)
        sys.exit(1)

    return

input_path = 'E:/Projekte/romp-test3/video_results.npz'
output_path = 'E:/Projekte/romp-test3/result.fbx'
subject_id = 0 # -1
rotate_y = True

print('Input path: ' + input_path)
print('Output path: ' + output_path)

startTime = time.perf_counter()

# Process pose file
poses_processed = process_poses(
    input_path=input_path,
    gender=gender,
    fps_source=fps_source,
    fps_target=fps_target,
    subject_id=subject_id
    )
export_animated_mesh(output_path, rotate_y)

print('--------------------------------------------------')
print('Animation export finished, save to ', output_path)
print('Poses processed: ', poses_processed)
print('Processing time : ', time.perf_counter() - startTime)
print('--------------------------------------------------')

Btw, now i can proof, that this line is indeed a bug: https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/simple_romp/tools/convert2fbx.py#L308

[vivi90]

@Arthur151 Okay i have run some tests with the latest full ROMP version (not simple-romp) and the Blender Addon.

In ROMP the romp.predict.webcam is sending it to blender 'as it is': https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/romp/predict/webcam.py#L130 https://github.com/Arthur151/ROMP/blob/1458ba0b05a63d64b7fff7898cd917d9fda378b5/romp/lib/visualization/socket_utils.py#L213 In Blender the plugin does all necessary calculations for the rig: https://github.com/yanch2116/CharacterDriven-BlenderAddon/blob/6c288acf8540f900202de10adc2463ab2ca75f04/src/characterDriven.py#L51

It's no problem to use an VRM rig instead the mixamo rig in Blender for example. For VMC i could implement it the easy way inside this blender script.

But how about moving the most of these calculations from Blender back to ROMP? There is another point: an VMC to Blender plugin still exists. So how about also replacing the visulize_platform of blender by vmc completely. Implementing VMC directly into ROMP makes it more flexible instead of using an proprietary blender connection.

[Arthur151]

Happy to see thing going so well with your efforts! Excellent work! I just finished another deadline. I don't know VMC well. Is there any support tools to implement VMC directly into ROMP?

[vivi90]

I just finished another deadline.

great! 🙂

[..] Is there any support tools to implement VMC directly into ROMP?

Currently there is no VMC protocol package for python. But since it's based on OSC, thats my drafts:

osc.py:

#!/usr/bin/env python3

from socket import gaierror
from osc4py3.as_eventloop import *
from osc4py3 import oscmethod as oscm
from osc4py3 import oscbuildparse as oscbp
import time

class Client:
    """Based on: https://github.com/barlaensdoonn/osc_basics/blob/master/osc_basics.py"""
    def __init__(self, host: str, port: int, name: str) -> None:
        self.host = host
        self.port = port
        self.name = name
        self.start()

    def __enter__(self) -> 'Client':
        return self

    def start(self):
        osc_startup()
        try:
            osc_udp_client(self.host, self.port, self.name)
            print("OSC client ready.")
        except gaierror:
            print(
                "Host '{}' unavailable, failed to create client '{}'.".format(
                    self.host, 
                    self.name
                )
            )
            raise

    def send(self, address: str, data_types: str, data: str) -> None:
        message = oscbp.OSCMessage(
            address,
            data_types,
            data
        )
        osc_send(
            message,
            self.name
        )
        osc_process()
        print(str(message))

    def send_bundle(self, address: str, data_types: str, data: str) -> None:
        message_bundle = []
        for data_tuple in data:
            message_bundle.append(
                oscbp.OSCMessage(
                    address,
                    data_types,
                    [
                        str(data_tuple[0]),
                        data_tuple[1].x,
                        data_tuple[1].y,
                        data_tuple[1].z,
                        data_tuple[2].x,
                        data_tuple[2].y,
                        data_tuple[2].z,
                        data_tuple[2].w
                    ]
                )
            )
        message_bundle = oscbp.OSCBundle(
            oscbp.unixtime2timetag(time.time()),
            message_bundle
        )
        osc_send(
            message_bundle,
            self.name
        )
        osc_process()
        print(str(message_bundle))

    def __exit__(self, type, value, traceback) -> None:
        self.__del__()

    def __del__(self) -> None:
        osc_terminate()
        print("OSC client down.")

vmc.py:

#!/usr/bin/env python3

from osc import Client as OSCClient
import time
from math import radians, degrees, cos, sin, atan2, asin, pow, floor

class Bone:
    valid_names = (
        "Hips",
        "LeftUpperLeg",
        "RightUpperLeg",
        "LeftLowerLeg",
        "RightLowerLeg",
        "LeftFoot",
        "RightFoot",
        "Spine",
        "Chest",
        "Neck",
        "Head",
        "LeftShoulder",
        "RightShoulder",
        "LeftUpperArm",
        "RightUpperArm",
        "LeftLowerArm",
        "RightLowerArm",
        "LeftHand",
        "RightHand",
        "LeftToes",
        "RightToes",
        "LeftEye",
        "RightEye",
        "Jaw",
        "LeftThumbProximal",
        "LeftThumbIntermediate",
        "LeftThumbDistal",
        "LeftIndexProximal",
        "LeftIndexIntermediate",
        "LeftIndexDistal",
        "LeftMiddleProximal",
        "LeftMiddleIntermediate",
        "LeftMiddleDistal",
        "LeftRingProximal",
        "LeftRingIntermediate",
        "LeftRingDistal",
        "LeftLittleProximal",
        "LeftLittleIntermediate",
        "LeftLittleDistal",
        "RightThumbProximal",
        "RightThumbIntermediate",
        "RightThumbDistal",
        "RightIndexProximal",
        "RightIndexIntermediate",
        "RightIndexDistal",
        "RightMiddleProximal",
        "RightMiddleIntermediate",
        "RightMiddleDistal",
        "RightRingProximal",
        "RightRingIntermediate",
        "RightRingDistal",
        "RightLittleProximal",
        "RightLittleIntermediate",
        "RightLittleDistal",
        "UpperChest"
    )

    def __init__(self, name: str) -> None:
        if name in self.valid_names:
            self.name = name
        else:
            raise ValueError("Invalid bone name.")

    def __str__(self) -> str:
        return self.name

class Position:
    def __init__(self, x: float, y: float, z: float) -> None:
        self.x = x
        self.y = y
        self.z = z

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z)
            )
        )

class Quaternion:
    def __init__(self, x: float, y: float, z: float, w: float) -> None:
        if round(pow(x, 2) + pow(y, 2) + pow(z, 2) + pow(w, 2), 1) == 1:
            self.x = x
            self.y = y
            self.z = z
            self.w = w
        else:
            raise ValueError(
                "Invalid quaternion values: x={}, y={}, z={}, w={}".format(
                    x,
                    y,
                    z,
                    w
                )
            )

    @classmethod
    def from_euler(cls, 
                   phi: float, theta: float, psi: float,
                   precision: int) -> 'Quaternion':
        """Creates an quaternion by euler angles and returns it

        Args:
            phi (float): Rotation angle around the X axis
            theta (float): Rotation angle around the Y axis
            psi (float): Rotation angle around the Y axis
            precision (int): Round the results to 'precision' digits after the decimal point

        Returns:
            Quaternion: Created quaternion (x, y, z, w)

        >>> str(Quaternion.from_euler(-90, -180, 90, 12))
        '(0.5, -0.5, -0.5, 0.5)'

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        cos_phi_half = cos(radians(phi) / 2)
        sin_phi_half = sin(radians(phi) / 2)
        # y axis rotation angle
        cos_theta_half = cos(radians(theta) / 2)
        sin_theta_half = sin(radians(theta) / 2)
        # z axis rotation angle
        cos_psi_half = cos(radians(psi) / 2)
        sin_psi_half = sin(radians(psi) / 2)
        # Calculation
        return cls(
            x = float(round(sin_phi_half * cos_theta_half * cos_psi_half - cos_phi_half * sin_theta_half * sin_psi_half, precision)),
            y = float(round(cos_phi_half * sin_theta_half * cos_psi_half + sin_phi_half * cos_theta_half * sin_psi_half, precision)),
            z = float(round(cos_phi_half * cos_theta_half * sin_psi_half - sin_phi_half * sin_theta_half * cos_psi_half, precision)),
            w = float(round(cos_phi_half * cos_theta_half * cos_psi_half + sin_phi_half * sin_theta_half * sin_psi_half, precision))
        )

    def to_euler(self) -> tuple[float, float, float]:
        """Exports an quaternion as an euler angle tuple

        Returns:
            tuple[float, float, float]: (x, y, z)

        >>> str(Quaternion.from_euler(-90, -180, 90, 12).to_euler())
        (90.0, 0.0, -90.0)

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        t0 = 2 * (self.w * self.x + self.y * self.z)
        t1 = 1 - 2 * (self.x * self.x + self.y * self.y)
        x = atan2(t0, t1)
        # y axis rotation angle
        t2 = 2 * (self.w * self.y - self.z * self.x)
        t2 = 1 if t2 > 1 else t2
        t2 = -1 if t2 < -1 else t2
        y = asin(t2)
        # y axis rotation angle
        t3 = 2 * (self.w * self.z + self.x * self.y)
        t4 = 1 - 2 * (self.y * self.y + self.z * self.z)
        z = atan2(t3, t4)
        return degrees(x), degrees(y), degrees(z)

    def __str__(self) -> str:
        return "(" + ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z), 
                str(self.w)
            )
        ) + ")"

class Assistant(OSCClient):
    def __init__(self, host: str, port: int, name: str) -> None:
        self.started_at = time.time()
        super().__init__(host, port, name)

    def send_root_transform(self, position: Position, 
                            quaternion: Quaternion) -> None:
        # since VMC v2.0.0
        self.send(
            "/VMC/Ext/Root/Pos",
            None,
            [
                "root",
                position.x,
                position.y,
                position.z, 
                quaternion.x,
                quaternion.y,
                quaternion.z,
                quaternion.w
            ]
        )

    def send_bones_transform(self,
                             transform: list[list[Bone, Position, Quaternion]]
                            )-> None:
        self.send_bundle(
            "/VMC/Ext/Bone/Pos",
            None,
            transform
        )

    def send_available_states(self, loaded: int, 
                              calibration_state: int = None,
                              calibration_mode: int = None,
                              tracking_status: int = None) -> None:
        if tracking_status == None:
            if calibration_state == None or calibration_mode == None:
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded
                    ]
                )
            else:
                # since VMC v2.5
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded,
                        calibration_state,
                        calibration_mode
                    ]
                )
        else:
            # since VMC v2.7
            self.send(
                "/VMC/Ext/OK",
                None,
                [
                    loaded,
                    calibration_state,
                    calibration_mode,
                    tracking_status
                ]
            )

    def send_relative_time(self) -> None:
        self.send(
            "/VMC/Ext/T",
            None,
            [
                float(time.time() - self.started_at)
            ]
        )

vrm.py:

#!/usr/bin/env python3

from gltflib import GLTF

class VRM(GLTF):
    def __init__(self, filename: str) -> None:
        gltf = super().load_glb(filename)
        super().__init__(gltf.model, gltf.resources)

    def export_to_glb(self, filename: str) -> None:
        super().export(filename)

    def export_to_gltf(self, filename: str) -> None:
        super().convert_to_file_resource(
            super().get_glb_resource(), 
            filename + ".resources.bin"
        )
        super().export(filename)

    def get_bones(self) -> list:
        return self.model.nodes

test.py:

#!/usr/bin/env python3

from log import Log
import sys
from vrm import VRM as Model
from vmc import Assistant as VMCAssistant, Quaternion
from gui import Window
import numpy as np
import romp, cv2

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
# settings.mode='video'
#print(settings)
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('test.jpg'))
print(outputs["smpl_thetas"][0].reshape(24, 3))

"""
# NPZ data
input_path = "E:/Projekte/romp-test/00000215.npz"
input_path = "video_results.npz"
frame_results = np.load(input_path, allow_pickle=True)['results'][()]
sequence_results = np.load(input_path, allow_pickle=True)['sequence_results'][()]
print(frame_results)
print(sequence_results)

# Avatar model
model = Model("test.vrm")
print(model.get_bones()[101]) # leftUpperLeg (see: https://github.com/vrm-c/vrm-specification/issues/371)
model_root = []
model_t_pose = []

# Quaternions example
q = Quaternion.from_euler(-90, -180, 90, precision = 12)
print(q)
e = q.to_euler()
print(e)

# GUI
Window(
    VMCAssistant(
        connection['host'],
        connection['port'],
        connection['name']
    ),
    model_root,
    model_t_pose
).MainLoop()
"""

gui.py:

#!/usr/bin/env python3

import wx
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion

class Window(wx.App):
    def __init__(self, vmc: VMCAssistant, 
                 model_root: list[Position, Quaternion],
                 model_t_pose: list[Bone, Position, Quaternion]) -> None:
        self.vmc = vmc
        self.model_root = model_root
        self.model_t_pose = model_t_pose
        super().__init__()

    def OnInit(self) -> bool:
        self.frame = self.Frame(
            vmc = self.vmc,
            parent = None, 
            title = 'VMC Assistant Demo', 
            size = wx.Size(400, 300)
        )
        self.frame.Show(True)
        return True

    class Frame(wx.Frame):
        def __init__(self, vmc: VMCAssistant, 
                     parent: wx.App, title: str, size: wx.Size) -> None:
            super().__init__(
                parent, 
                title = title, 
                size = size
            )
            self.panel = self.Panel(self, vmc)

        class Panel(wx.Panel):

            def __init__(self, parent: wx.Frame, vmc: VMCAssistant) -> None:
                self.vmc = vmc
                super().__init__(parent)
                # Controls
                self.test_caption = wx.StaticText(
                    self, label = 'Test', 
                    style = wx.ALIGN_LEFT
                )
                self.test_slider = wx.Slider(
                    self, 
                    value = 0, 
                    minValue = -180, 
                    maxValue = 180, 
                    style = wx.SL_HORIZONTAL | wx.SL_LABELS
                )
                self.Bind(wx.EVT_SLIDER, self.change_test)
                # Log
                self.log = wx.TextCtrl(
                    self, 
                    style = wx.TE_MULTILINE | wx.TE_READONLY
                )
                # Layout
                self.layout = wx.BoxSizer(wx.VERTICAL)
                self.layout.AddMany([
                    (self.test_caption, 0, wx.EXPAND | wx.ALIGN_LEFT | wx.ALL, 5),
                    (self.test_slider, 0, wx.EXPAND | wx.ALL, 5),
                    (self.log, 1, wx.EXPAND | wx.ALIGN_LEFT | wx.ALL, 5)
                ])
                self.SetSizer(self.layout)
                self.Center()

            def change_test(self, event: wx.Event) -> None:
                value = float(self.test_slider.GetValue())
                quat = Quaternion.from_euler(0.0, 0.0, value, 12)
                self.vmc.send_bones_transform(
                    [
                        [
                            Bone("LeftUpperLeg"),
                            Position(0.0, 0.0, 0.0),
                            quat
                        ]
                    ]
                )
                self.vmc.send_available_states(1)
                self.vmc.send_relative_time()
                self.log.AppendText(
                    str(quat) + "\r\n"
                )

[Arthur151]

Great job. It seems interesting~ We need to install some additional packages, instead of blender to drive other virtual avatars. Looks great. I'd like to join your project.

[Arthur151]

Please let me know if you want me to add / check BEV/ROMP features to it. I think that people may love it for its convenience.

[vivi90]

[..] I'd like to join your project.

Already invited you: https://github.com/vivi90/python-vmc/invitations

Please let me know if you want me to add / check BEV/ROMP features to it. [..]

What do you mean by features?

Currently i can only find the output for the 24 SMPL joints. In the README of simple-romp are also another joints for face and hands mentioned:

...
SMPL_EXTRA_30 = {
'Nose':24, 'R_Eye':25, 'L_Eye':26, 'R_Ear': 27, 'L_Ear':28,
'L_BigToe':29, 'L_SmallToe': 30, 'L_Heel':31, 'R_BigToe':32,'R_SmallToe':33, 'R_Heel':34,
'L_Hand_thumb':35, 'L_Hand_index': 36, 'L_Hand_middle':37, 'L_Hand_ring':38, 'L_Hand_pinky':39,
'R_Hand_thumb':40, 'R_Hand_index':41,'R_Hand_middle':42, 'R_Hand_ring':43, 'R_Hand_pinky': 44,
'R_Hip': 45, 'L_Hip':46, 'Neck_LSP':47, 'Head_top':48, 'Pelvis':49, 'Thorax_MPII':50,
'Spine_H36M':51, 'Jaw_H36M':52, 'Head':53}

Would be great to have also these joints available.

[Arthur151]

Hi, all 54(24+30) joints are saved as 'j3d_all54' in output .npz file of originial ROMP, see this 'joints' in output .npz file of simple-romp.

[vivi90]

Hi, all 54(24+30) joints are saved as 'j3d_all54' in output .npz file of originial ROMP, see this 'joints' in output .npz file of simple-romp.

Thank you! 😄

[vivi90]

@Arthur151 I have progress! 😃 vmc.py:

 #!/usr/bin/env python3

from osc import Client as OSCClient
import time
from math import radians, degrees, cos, sin, atan2, asin, pow, floor

class Bone:
    # 55 VRM bones
    valid_names = (
        "Hips",
        "LeftUpperLeg",
        "RightUpperLeg",
        "LeftLowerLeg",
        "RightLowerLeg",
        "LeftFoot",
        "RightFoot",
        "Spine",
        "Chest",
        "Neck",
        "Head",
        "LeftShoulder",
        "RightShoulder",
        "LeftUpperArm",
        "RightUpperArm",
        "LeftLowerArm",
        "RightLowerArm",
        "LeftHand",
        "RightHand",
        "LeftToes",
        "RightToes",
        "LeftEye",
        "RightEye",
        "Jaw",
        "LeftThumbProximal",
        "LeftThumbIntermediate",
        "LeftThumbDistal",
        "LeftIndexProximal",
        "LeftIndexIntermediate",
        "LeftIndexDistal",
        "LeftMiddleProximal",
        "LeftMiddleIntermediate",
        "LeftMiddleDistal",
        "LeftRingProximal",
        "LeftRingIntermediate",
        "LeftRingDistal",
        "LeftLittleProximal",
        "LeftLittleIntermediate",
        "LeftLittleDistal",
        "RightThumbProximal",
        "RightThumbIntermediate",
        "RightThumbDistal",
        "RightIndexProximal",
        "RightIndexIntermediate",
        "RightIndexDistal",
        "RightMiddleProximal",
        "RightMiddleIntermediate",
        "RightMiddleDistal",
        "RightRingProximal",
        "RightRingIntermediate",
        "RightRingDistal",
        "RightLittleProximal",
        "RightLittleIntermediate",
        "RightLittleDistal",
        "UpperChest"
    )

    def __init__(self, name: str) -> None:
        if name in self.valid_names:
            self.name = name
        else:
            raise ValueError("Invalid bone name.")

    def __str__(self) -> str:
        return self.name

class Position:
    def __init__(self, x: float, y: float, z: float) -> None:
        self.x = float(x)
        self.y = float(y)
        self.z = float(z)

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z)
            )
        )

class Quaternion:
    def __init__(self, x: float, y: float, z: float, w: float) -> None:
        if round(pow(x, 2) + pow(y, 2) + pow(z, 2) + pow(w, 2), 1) == 1:
            self.x = float(x)
            self.y = float(y)
            self.z = float(z)
            self.w = float(w)
        else:
            raise ValueError(
                "Invalid quaternion values: x={}, y={}, z={}, w={}".format(
                    x,
                    y,
                    z,
                    w
                )
            )

    @classmethod
    def from_euler(cls, 
                   phi: float, theta: float, psi: float,
                   precision: int) -> 'Quaternion':
        """Creates an quaternion by euler angles and returns it

        Args:
            phi (float): Rotation angle around the X axis
            theta (float): Rotation angle around the Y axis
            psi (float): Rotation angle around the Y axis
            precision (int): Round the results to 'precision' digits after the decimal point

        Returns:
            Quaternion: Created quaternion (x, y, z, w)

        >>> str(Quaternion.from_euler(-90, -180, 90, 12))
        '(0.5, -0.5, -0.5, 0.5)'

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        cos_phi_half = cos(radians(phi) / 2)
        sin_phi_half = sin(radians(phi) / 2)
        # y axis rotation angle
        cos_theta_half = cos(radians(theta) / 2)
        sin_theta_half = sin(radians(theta) / 2)
        # z axis rotation angle
        cos_psi_half = cos(radians(psi) / 2)
        sin_psi_half = sin(radians(psi) / 2)
        # Calculation
        return cls(
            x = float(round(sin_phi_half * cos_theta_half * cos_psi_half - cos_phi_half * sin_theta_half * sin_psi_half, precision)),
            y = float(round(cos_phi_half * sin_theta_half * cos_psi_half + sin_phi_half * cos_theta_half * sin_psi_half, precision)),
            z = float(round(cos_phi_half * cos_theta_half * sin_psi_half - sin_phi_half * sin_theta_half * cos_psi_half, precision)),
            w = float(round(cos_phi_half * cos_theta_half * cos_psi_half + sin_phi_half * sin_theta_half * sin_psi_half, precision))
        )

    def to_euler(self) -> tuple[float, float, float]:
        """Exports an quaternion as an euler angle tuple

        Returns:
            tuple[float, float, float]: (x, y, z)

        >>> str(Quaternion.from_euler(-90, -180, 90, 12).to_euler())
        (90.0, 0.0, -90.0)

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        t0 = 2 * (self.w * self.x + self.y * self.z)
        t1 = 1 - 2 * (self.x * self.x + self.y * self.y)
        x = atan2(t0, t1)
        # y axis rotation angle
        t2 = 2 * (self.w * self.y - self.z * self.x)
        t2 = 1 if t2 > 1 else t2
        t2 = -1 if t2 < -1 else t2
        y = asin(t2)
        # y axis rotation angle
        t3 = 2 * (self.w * self.z + self.x * self.y)
        t4 = 1 - 2 * (self.y * self.y + self.z * self.z)
        z = atan2(t3, t4)
        return degrees(x), degrees(y), degrees(z)

    def __str__(self) -> str:
        return "(" + ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z), 
                str(self.w)
            )
        ) + ")"

class Assistant(OSCClient):
    def __init__(self, host: str, port: int, name: str) -> None:
        self.started_at = time.time()
        super().__init__(host, port, name)

    def send_root_transform(self, position: Position, 
                            quaternion: Quaternion) -> None:
        # since VMC v2.0.0
        self.send(
            "/VMC/Ext/Root/Pos",
            None,
            [
                "root",
                position.x,
                position.y,
                position.z, 
                quaternion.x,
                quaternion.y,
                quaternion.z,
                quaternion.w
            ]
        )

    def send_bones_transform(self,
                             transform: list[list[Bone, Position, Quaternion]]
                            )-> None:
        self.send_bundle(
            "/VMC/Ext/Bone/Pos",
            None,
            transform
        )

    def send_available_states(self, loaded: int, 
                              calibration_state: int = None,
                              calibration_mode: int = None,
                              tracking_status: int = None) -> None:
        if tracking_status == None:
            if calibration_state == None or calibration_mode == None:
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded
                    ]
                )
            else:
                # since VMC v2.5
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded,
                        calibration_state,
                        calibration_mode
                    ]
                )
        else:
            # since VMC v2.7
            self.send(
                "/VMC/Ext/OK",
                None,
                [
                    loaded,
                    calibration_state,
                    calibration_mode,
                    tracking_status
                ]
            )

    def send_relative_time(self) -> None:
        self.send(
            "/VMC/Ext/T",
            None,
            [
                float(time.time() - self.started_at)
            ]
        )

romp_test.sh:

#!/usr/bin/env python3

from log import Log
import sys
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion
import numpy as np
import romp, cv2
from scipy.spatial.transform import Rotation as r

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_rotations_by_axis = outputs["smpl_thetas"][0].reshape(24, 3)

# ROMP
settings = romp.main.default_settings
# settings.mode='video'
subject_id = 0
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_root_position = outputs["cam_trans"][subject_id]
smpl_rotations_by_axis = outputs["smpl_thetas"][subject_id].reshape(24, 3)

# Example
rotation_by_axis = smpl_rotations_by_axis[0]
rotation_by_quaternion = r.from_rotvec(rotation_by_axis).as_quat()
print("ROMP settings: " + str(settings))
print("Root position: " + str(smpl_root_position))
print("Rotation by axis: " + str(rotation_by_axis))
print("Rotation by quaternion: " + str(rotation_by_quaternion))

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips",
    1 : "LeftUpperLeg",
    2 : "RightUpperLeg",
    3 : "Spine",
    4 : "LeftLowerLeg",
    5 : "RightLowerLeg",
    6 : "Chest",
    7 : "LeftFoot",
    8 : "RightFoot",
    9 : "UpperChest",
    10 : "LeftToes",
    11 : "RightToes",
    12 : "Neck",
    13 : "LeftIndexProximal",
    14 : "RightIndexProximal",
    15 : "Head",
    16 : "LeftShoulder",
    17 : "RightShoulder",
    18 : "LeftUpperArm",
    19 : "RightUpperArm",
    20 : "LeftLowerArm",
    21 : "RightLowerArm",
    22 : "LeftHand",
    23 : "RightHand"
}

# VMC
vmc = VMCAssistant(connection['host'], connection['port'], connection['name'])
bones = []
for index, rotation in enumerate(smpl_rotations_by_axis):
    rotation = r.from_rotvec(rotation).as_quat()
    bones.append(
        [
            Bone(vrm_bone_names[index]),
            Position(0.0, 0.0, 0.0),
            Quaternion(
                rotation[0], 
                rotation[1],
                rotation[2], 
                rotation[3],
            )
        ]
    )

bones += [
    [Bone("LeftEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("Jaw"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleProximal"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)]
]
vmc.send_root_transform(
    Position(
        smpl_root_position[0], 
        smpl_root_position[1],
        smpl_root_position[2]
    ), 
    Quaternion(0, 0, 0, 1)
)
vmc.send_bones_transform(bones)
vmc.send_available_states(1)
vmc.send_relative_time()

The result is not perfect (a lot of joints not working and the root rotation was a little bit wrong) but i am confident to be able to fix it easily soon.

[Arthur151]

Congrats! Really happy to see your rapid progress. Let me know if you need my help.

[vivi90]

@Arthur151 Yes, i will, thank you 🙂

At first i have to fix some VMC protocol communication issues (the receiver sometimes ignores the sender and resetting the pose). (just writing it here to document it) https://github.com/emilianavt/VSeeFaceReleases/issues/15

[vivi90]

@Arthur151 I fixed some mapping mistakes, i made:

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    13 : "LeftShoulder", # L_Collar
    14 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    16 : "LeftUpperArm", # L_Shoulder
    17 : "RightUpperArm", # R_Shoulder
    18 : "LeftLowerArm", # L_Elbow
    19 : "RightLowerArm", # R_Elbow
    20 : "LeftHand", # L_Wrist
    21 : "RightHand", # R_Wrist
    22 : "LeftMiddleProximal", # L_Hand
    23 : "RightMiddleProximal" # R_hand
}

But aside from communication issues, there are very unlogical rotation issues: Also until now i am not able to fix the wrong pose of the arms. 🙁 See the source image: vmc.py:

#!/usr/bin/env python3

from osc import Client as OSCClient
import time
from math import radians, degrees, cos, sin, atan2, asin, pow, floor

class Bone:
    # 55 VRM bones
    valid_names = (
        "Hips",
        "LeftUpperLeg",
        "RightUpperLeg",
        "LeftLowerLeg",
        "RightLowerLeg",
        "LeftFoot",
        "RightFoot",
        "Spine",
        "Chest",
        "Neck",
        "Head",
        "LeftShoulder",
        "RightShoulder",
        "LeftUpperArm",
        "RightUpperArm",
        "LeftLowerArm",
        "RightLowerArm",
        "LeftHand",
        "RightHand",
        "LeftToes",
        "RightToes",
        "LeftEye",
        "RightEye",
        "Jaw",
        "LeftThumbProximal",
        "LeftThumbIntermediate",
        "LeftThumbDistal",
        "LeftIndexProximal",
        "LeftIndexIntermediate",
        "LeftIndexDistal",
        "LeftMiddleProximal",
        "LeftMiddleIntermediate",
        "LeftMiddleDistal",
        "LeftRingProximal",
        "LeftRingIntermediate",
        "LeftRingDistal",
        "LeftLittleProximal",
        "LeftLittleIntermediate",
        "LeftLittleDistal",
        "RightThumbProximal",
        "RightThumbIntermediate",
        "RightThumbDistal",
        "RightIndexProximal",
        "RightIndexIntermediate",
        "RightIndexDistal",
        "RightMiddleProximal",
        "RightMiddleIntermediate",
        "RightMiddleDistal",
        "RightRingProximal",
        "RightRingIntermediate",
        "RightRingDistal",
        "RightLittleProximal",
        "RightLittleIntermediate",
        "RightLittleDistal",
        "UpperChest"
    )

    def __init__(self, name: str) -> None:
        if name in self.valid_names:
            self.name = name
        else:
            raise ValueError("Invalid bone name.")

    def __str__(self) -> str:
        return self.name

class Position:
    def __init__(self, x: float, y: float, z: float) -> None:
        self.x = float(x)
        self.y = float(y)
        self.z = float(z)

    @classmethod
    def identity(cls) -> 'Position':
        """Creates an identity position and returns it

        Returns:
            Position: Created identity position (x, y, z, w)

        >>> Position.identity()
        0.0, 0.0, 0.0

        """
        return cls(
            x = float(0.0),
            y = float(0.0),
            z = float(0.0)
        )

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z)
            )
        )

class Quaternion:
    def __init__(self, x: float, y: float, z: float, w: float) -> None:
        if round(pow(x, 2) + pow(y, 2) + pow(z, 2) + pow(w, 2), 1) == 1:
            self.x = float(x)
            self.y = float(y)
            self.z = float(z)
            self.w = float(w)
        else:
            raise ValueError(
                "Invalid quaternion values: x={}, y={}, z={}, w={}".format(
                    x,
                    y,
                    z,
                    w
                )
            )

    @classmethod
    def identity(cls) -> 'Quaternion':
        """Creates an identity quaternion and returns it

        Returns:
            Quaternion: Created identity quaternion (x, y, z, w)

        >>> Quaternion.identity()
        0.0, 0.0, 0.0, 1.0

        """
        return cls(
            x = float(0.0),
            y = float(0.0),
            z = float(0.0),
            w = float(1.0)
        )

    @classmethod
    def from_euler(cls, 
                   phi: float, theta: float, psi: float,
                   precision: int) -> 'Quaternion':
        """Creates an quaternion by euler angles and returns it

        Args:
            phi (float): Rotation angle around the X axis
            theta (float): Rotation angle around the Y axis
            psi (float): Rotation angle around the Y axis
            precision (int): Round the results to 'precision' digits after the decimal point

        Returns:
            Quaternion: Created quaternion (x, y, z, w)

        >>> Quaternion.from_euler(-90, -180, 90, 12)
        0.5, -0.5, -0.5, 0.5

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        cos_phi_half = cos(radians(phi) / 2)
        sin_phi_half = sin(radians(phi) / 2)
        # y axis rotation angle
        cos_theta_half = cos(radians(theta) / 2)
        sin_theta_half = sin(radians(theta) / 2)
        # z axis rotation angle
        cos_psi_half = cos(radians(psi) / 2)
        sin_psi_half = sin(radians(psi) / 2)
        # Calculation
        return cls(
            x = float(round(sin_phi_half * cos_theta_half * cos_psi_half - cos_phi_half * sin_theta_half * sin_psi_half, precision)),
            y = float(round(cos_phi_half * sin_theta_half * cos_psi_half + sin_phi_half * cos_theta_half * sin_psi_half, precision)),
            z = float(round(cos_phi_half * cos_theta_half * sin_psi_half - sin_phi_half * sin_theta_half * cos_psi_half, precision)),
            w = float(round(cos_phi_half * cos_theta_half * cos_psi_half + sin_phi_half * sin_theta_half * sin_psi_half, precision))
        )

    def to_euler(self) -> tuple[float, float, float]:
        """Exports an quaternion as an euler angle tuple

        Returns:
            tuple[float, float, float]: (x, y, z)

        >>> Quaternion.from_euler(-90, -180, 90, 12).to_euler()
        90.0, 0.0, -90.0

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        t0 = 2 * (self.w * self.x + self.y * self.z)
        t1 = 1 - 2 * (self.x * self.x + self.y * self.y)
        x = atan2(t0, t1)
        # y axis rotation angle
        t2 = 2 * (self.w * self.y - self.z * self.x)
        t2 = 1 if t2 > 1 else t2
        t2 = -1 if t2 < -1 else t2
        y = asin(t2)
        # y axis rotation angle
        t3 = 2 * (self.w * self.z + self.x * self.y)
        t4 = 1 - 2 * (self.y * self.y + self.z * self.z)
        z = atan2(t3, t4)
        return degrees(x), degrees(y), degrees(z)

    def multiply_by(self, quaternion: 'Quaternion',  precision: int) -> 'Quaternion':
        """Quaternion multiplication

        Returns:
            Quaternion: Multiplication product quaternion (x, y, z, w)

        >>> Quaternion.identity().multiply_by(Quaternion.from_euler(90, 0, 0, 12), 12)
        0.707106781187, 0.0, 0.0, 0.707106781187

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # Calculation
        return Quaternion(
            x = float(round(self.w * quaternion.x + self.x * quaternion.w + self.y * quaternion.z - self.z * quaternion.y, precision)),
            y = float(round(self.w * quaternion.y + self.y * quaternion.w + self.z * quaternion.x - self.x * quaternion.z, precision)),
            z = float(round(self.w * quaternion.z + self.z * quaternion.w + self.x * quaternion.y - self.y * quaternion.x, precision)),
            w = float(round(self.w * quaternion.w - self.x * quaternion.x - self.y * quaternion.y - self.z * quaternion.z, precision))
        )

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z), 
                str(self.w)
            )
        )

class Assistant(OSCClient):
    def __init__(self, host: str, port: int, name: str) -> None:
        self.started_at = time.time()
        super().__init__(host, port, name)

    def send_root_transform(self, position: Position, 
                            quaternion: Quaternion) -> None:
        # since VMC v2.0.0
        self.send(
            "/VMC/Ext/Root/Pos",
            None,
            [
                "root",
                position.x,
                position.y,
                position.z, 
                quaternion.x,
                quaternion.y,
                quaternion.z,
                quaternion.w
            ]
        )

    def send_bones_transform(self,
                             transform: list[list[Bone, Position, Quaternion]]
                            )-> None:
        self.send_bundle(
            "/VMC/Ext/Bone/Pos",
            None,
            transform
        )

    def send_available_states(self, loaded: int, 
                              calibration_state: int = None,
                              calibration_mode: int = None,
                              tracking_status: int = None) -> None:
        if tracking_status == None:
            if calibration_state == None or calibration_mode == None:
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded
                    ]
                )
            else:
                # since VMC v2.5
                self.send(
                    "/VMC/Ext/OK",
                    None,
                    [
                        loaded,
                        calibration_state,
                        calibration_mode
                    ]
                )
        else:
            # since VMC v2.7
            self.send(
                "/VMC/Ext/OK",
                None,
                [
                    loaded,
                    calibration_state,
                    calibration_mode,
                    tracking_status
                ]
            )

    def send_relative_time(self) -> None:
        self.send(
            "/VMC/Ext/T",
            None,
            [
                float(time.time() - self.started_at)
            ]
        )

romp_test.py:

#!/usr/bin/env python3

from log import Log
import sys
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion
import numpy as np
import romp, cv2
from scipy.spatial.transform import Rotation as r

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
subject_id = 0
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_root_position = outputs["cam_trans"][subject_id]
smpl_rotations_by_axis = outputs["smpl_thetas"][subject_id].reshape(24, 3)

# Info
rotation_by_axis = smpl_rotations_by_axis[0] # Pelvis
rotation_by_quaternion = r.from_rotvec(rotation_by_axis).as_quat()
print("ROMP settings: " + str(settings))
print("Root position: " + str(smpl_root_position))
print("Pelvis rotation by axis: " + str(rotation_by_axis))
print("Pelvis rotation by quaternion: " + str(rotation_by_quaternion))

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    13 : "LeftShoulder", # L_Collar
    14 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    16 : "LeftUpperArm", # L_Shoulder
    17 : "RightUpperArm", # R_Shoulder
    18 : "LeftLowerArm", # L_Elbow
    19 : "RightLowerArm", # R_Elbow
    20 : "LeftHand", # L_Wrist
    21 : "RightHand", # R_Wrist
    22 : "LeftMiddleProximal", # L_Hand
    23 : "RightMiddleProximal" # R_hand
}

# VMC
vmc = VMCAssistant(connection['host'], connection['port'], connection['name'])
bones = []
for index, rotation in enumerate(smpl_rotations_by_axis):
    rotation = r.from_rotvec(rotation).as_quat()
    bones.append(
        [
            Bone(vrm_bone_names[index]),
            Position(0.0, 0.0, 0.0),
            Quaternion(
                rotation[0], 
                rotation[1],
                rotation[2], 
                rotation[3],
            ).multiply_by(Quaternion.from_euler(0, 0, 0, 12), 12)
        ]
    )

bones += [
    [Bone("LeftEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("Jaw"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("LeftMiddleProximal"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("RightMiddleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)]
]
vmc.send_root_transform(
    Position(smpl_root_position[0], smpl_root_position[1], smpl_root_position[2]),
    bones[0][2].multiply_by(Quaternion.from_euler(0, 90, 0, 12), 12) # Hips / Pelvis rotation
)
vmc.send_bones_transform(bones)
vmc.send_available_states(1)
vmc.send_relative_time()

log.txt:

2022-06-13T23:35:37.298172+02:00 [INFO] Using ROMP v1
2022-06-13T23:35:38.698688+02:00 [INFO] ROMP 'forward' executed in 0.6995s, FPS 1.4
2022-06-13T23:35:38.698688+02:00 [INFO] ROMP settings: Namespace(mode='image', input=None, save_path='C:\\Users\\Vivien\\ROMP_results', GPU=-1, onnx=False, temporal_optimize=False, center_thresh=0.25, show_largest=False, smooth_coeff=3.0, calc_smpl=True, render_mesh=False, renderer='sim3dr', show=False, show_items='mesh', save_video=False, frame_rate=24, smpl_path='C:\\Users\\Vivien\\.romp\\smpl_packed_info.pth', model_path='C:\\Users\\Vivien\\.romp\\ROMP.pkl', model_onnx_path='C:\\Users\\Vivien\\.romp\\ROMP.onnx', root_align=False)
2022-06-13T23:35:38.698688+02:00 [INFO] Root position: [-0.01749535  0.3001184   1.3419644 ]
2022-06-13T23:35:38.699687+02:00 [INFO] Pelvis rotation by axis: [-2.3814533  0.2776083 -1.8047591]
2022-06-13T23:35:38.699687+02:00 [INFO] Pelvis rotation by quaternion: [-0.7916114   0.0922789  -0.59991432  0.07027662]
2022-06-13T23:35:38.706687+02:00 [INFO] OSC client ready.
2022-06-13T23:35:38.708687+02:00 [INFO] OSCMessage(addrpattern='/VMC/Ext/Root/Pos', typetags=None, arguments=['root', -0.017495352774858475, 0.3001183867454529, 1.3419643640518188, -0.135550308314, 0.114944112396, -0.983957272769, -0.015557966077])
2022-06-13T23:35:38.686687+02:00 [ERROR] C:\tools\miniconda3\envs\vmc\lib\site-packages\romp\post_parser.py:34: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  topk_ys = (topk_inds.long() // w).float()
2022-06-13T23:35:38.686687+02:00 [ERROR] C:\tools\miniconda3\envs\vmc\lib\site-packages\romp\post_parser.py:39: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  topk_clses = index.long() // K
2022-06-13T23:35:38.690687+02:00 [ERROR] C:\tools\miniconda3\envs\vmc\lib\site-packages\romp\post_parser.py:144: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
  parsed_results['center_preds'] = torch.stack([flat_inds%64, flat_inds//64],1) * 512 // 64
2022-06-13T23:35:38.709687+02:00 [INFO] OSCBundle(timetag=OSCtimetag(sec=3864144938, frac=3043786752), elements=[OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Hips', 0.0, 0.0, 0.0, -0.791611402173, 0.092278904647, -0.599914317772, 0.070276618018]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftUpperLeg', 0.0, 0.0, 0.0, -0.569858013359, -0.06745372591, 0.217820886161, 0.789471912751]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightUpperLeg', 0.0, 0.0, 0.0, 0.090121532945, -0.081781225552, -0.206500702702, 0.970848804001]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Spine', 0.0, 0.0, 0.0, 0.159540863334, 0.056978387243, -0.01746822004, 0.985390804505]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftLowerLeg', 0.0, 0.0, 0.0, 0.515684824837, 0.071990186804, -0.080458957665, 0.849948781144]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightLowerLeg', 0.0, 0.0, 0.0, 0.643227349666, 0.124634888021, -0.038055220216, 0.754504156081]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Chest', 0.0, 0.0, 0.0, -0.009237677233, -0.004677920042, -0.015887118318, 0.999820174759]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftFoot', 0.0, 0.0, 0.0, -0.014397013277, 0.055690825319, -0.118324337099, 0.991307525057]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightFoot', 0.0, 0.0, 0.0, -0.105811762628, -0.046522453533, 0.08669152996, 0.989507003937]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['UpperChest', 0.0, 0.0, 0.0, -0.014591640423, 0.038391712481, -0.009008571938, 0.999115611966]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftToes', 0.0, 0.0, 0.0, -0.143894841354, 0.097467170231, 0.230115105272, 0.95751838817]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightToes', 0.0, 0.0, 0.0, -0.172622070362, -0.052420857208, -0.118253663381, 0.976457754156]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Neck', 0.0, 0.0, 0.0, -0.088078933892, -0.010544569572, -0.009392014262, 0.996013405294]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftShoulder', 0.0, 0.0, 0.0, -0.073637693785, -0.1423213649, -0.026770841538, 0.986714467914]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightShoulder', 0.0, 0.0, 0.0, -0.062228765822, 0.192119945696, 0.015868997275, 0.97926793172]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Head', 0.0, 0.0, 0.0, 0.038323559786, -0.02462398172, -0.00271254098, 0.998958260595]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftUpperArm', 0.0, 0.0, 0.0, -0.054425329707, -0.323934342437, -0.309805007604, 0.892258528981]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightUpperArm', 0.0, 0.0, 0.0, -0.101450968563, 0.308201854737, 0.25927482901, 0.909668005789]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftLowerArm', 0.0, 0.0, 0.0, -0.007941600439, -0.785271786745, 0.335391734456, 0.520382106133]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightLowerArm', 0.0, 0.0, 0.0, -0.094142196633, 0.679053855173, -0.279761193504, 0.672128546632]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftHand', 0.0, 0.0, 0.0, -0.076182735673, -0.072401241884, 0.112182688038, 0.988114009345]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightHand', 0.0, 0.0, 0.0, -0.049100699661, 0.054419779377, -0.094317520884, 0.992840276258]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftMiddleProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightMiddleProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftEye', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightEye', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['Jaw', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftThumbProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftThumbIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftThumbDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftIndexProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftIndexIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftIndexDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftMiddleIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftMiddleDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftRingProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftRingIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftRingDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftLittleProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftLittleIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['LeftLittleDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightThumbProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightThumbIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightThumbDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightIndexProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightIndexIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightIndexDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightMiddleIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightMiddleDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightRingProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightRingIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightRingDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightLittleProximal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightLittleIntermediate', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0]), OSCMessage(addrpattern='/VMC/Ext/Bone/Pos', typetags=None, arguments=['RightLittleDistal', 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0])])
2022-06-13T23:35:38.710687+02:00 [INFO] OSCMessage(addrpattern='/VMC/Ext/OK', typetags=None, arguments=[1])
2022-06-13T23:35:38.710687+02:00 [INFO] OSCMessage(addrpattern='/VMC/Ext/T', typetags=None, arguments=[0.010999917984008789])

Any ideas, how to fix it?

[Arthur151]

Hi, I think there might be two reasons that cause this problem. I think the mapping between the shoulder joints might be wrong.

1. About the joint mapping:

   13 : "LeftShoulder", # L_Collar
   14 : "RightShoulder", # R_Collar

   Are you sure that the Shoulders correpsond to the Collar, instead of the Shoulders of SMPL model? Maybe correct mappsing is Collar + Shoulders of smpl parameters -> Shoulders of avatar. Maybe we need to re-calcuate the shoulders' 3D rotations using joint location.

2. If not the problem introduced above, then the 3D coordinate of shoulder for this avatar might rotate 180 degree in x/y axis, compared with the one of smpl shoulders. To fix it, you need to apply an 180 rotation in x/y to the smpl shoulder parameters.

[vivi90]

@Arthur151

[..] the mapping between the shoulder joints might be wrong. [..]

13 : "LeftShoulder", # L_Collar
14 : "RightShoulder", # R_Collar

Are you sure that the Shoulders correpsond to the Collar, instead of the Shoulders of SMPL model?

Not 100% sure but after comparing both models (SMPL and VRM) in Blender, it seems to be suitable.

Maybe correct mappsing is Collar + Shoulders of smpl parameters -> Shoulders of avatar. Maybe we need to re-calcuate the shoulders' 3D rotations using joint location.

hm, which way do you have in mind?

2. If not the problem introduced above, then the 3D coordinate of shoulder for this avatar might rotate 180 degree in x/y axis, compared with the one of smpl shoulders. To fix it, you need to apply an 180 rotation in x/y to the smpl shoulder parameters.

Tested some 180 degree rotations. Unfortunately i have not taken pictures of it. But even is it seems to be a way more better rotation, the bending of the mesh looked weird. Don't know why.

In the meanwhile i have uploaded my current draft: https://github.com/vivi90/python-vmc and tested some other ideas.

Swapping Swapped LeftShoulder, LeftUpperArm, LeftLowerArm, LeftHand, LeftMiddleProximal, RightShoulder, RightUpperArm, RightLowerArm, RightHand, RightMiddleProximal.

Rotation conjugation Conjugated LeftShoulder, LeftUpperArm, LeftLowerArm, LeftHand, LeftMiddleProximal, RightShoulder, RightUpperArm, RightLowerArm, RightHand, RightMiddleProximal.

But to be honest, i am not sure, if my mathematical calculations are 100% correct. Quaternion class in vmc.py:

#!/usr/bin/env python3

from math import radians, degrees, cos, sin, atan2, asin, pow, floor

class Quaternion:
    def __init__(self, x: float, y: float, z: float, w: float) -> None:
        if round(pow(x, 2) + pow(y, 2) + pow(z, 2) + pow(w, 2), 1) == 1:
            self.x = float(x)
            self.y = float(y)
            self.z = float(z)
            self.w = float(w)
        else:
            raise ValueError(
                "Invalid quaternion values: x={}, y={}, z={}, w={}".format(
                    x,
                    y,
                    z,
                    w
                )
            )

    @classmethod
    def identity(cls) -> 'Quaternion':
        """Creates an identity quaternion and returns it

        Returns:
            Quaternion: Created identity quaternion (x, y, z, w)

        >>> Quaternion.identity()
        0.0, 0.0, 0.0, 1.0

        """
        return cls(
            x = float(0.0),
            y = float(0.0),
            z = float(0.0),
            w = float(1.0)
        )

    @classmethod
    def from_euler(cls, 
                   phi: float, theta: float, psi: float,
                   precision: int) -> 'Quaternion':
        """Creates an quaternion by euler angles and returns it

        Args:
            phi (float): Rotation angle around the X axis
            theta (float): Rotation angle around the Y axis
            psi (float): Rotation angle around the Y axis
            precision (int): Round the results to 'precision' digits after the decimal point

        Returns:
            Quaternion: Created quaternion (x, y, z, w)

        >>> Quaternion.from_euler(-90, -180, 90, 12)
        0.5, -0.5, -0.5, 0.5

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        cos_phi_half = cos(radians(phi) / 2)
        sin_phi_half = sin(radians(phi) / 2)
        # y axis rotation angle
        cos_theta_half = cos(radians(theta) / 2)
        sin_theta_half = sin(radians(theta) / 2)
        # z axis rotation angle
        cos_psi_half = cos(radians(psi) / 2)
        sin_psi_half = sin(radians(psi) / 2)
        # Calculation
        return cls(
            x = float(round(sin_phi_half * cos_theta_half * cos_psi_half - cos_phi_half * sin_theta_half * sin_psi_half, precision)),
            y = float(round(cos_phi_half * sin_theta_half * cos_psi_half + sin_phi_half * cos_theta_half * sin_psi_half, precision)),
            z = float(round(cos_phi_half * cos_theta_half * sin_psi_half - sin_phi_half * sin_theta_half * cos_psi_half, precision)),
            w = float(round(cos_phi_half * cos_theta_half * cos_psi_half + sin_phi_half * sin_theta_half * sin_psi_half, precision))
        )

    def to_euler(self) -> tuple[float, float, float]:
        """Exports an quaternion as an euler angle tuple

        Returns:
            tuple[float, float, float]: (x, y, z)

        >>> Quaternion.from_euler(-90, -180, 90, 12).to_euler()
        90.0, 0.0, -90.0

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        t0 = 2 * (self.w * self.x + self.y * self.z)
        t1 = 1 - 2 * (self.x * self.x + self.y * self.y)
        x = atan2(t0, t1)
        # y axis rotation angle
        t2 = 2 * (self.w * self.y - self.z * self.x)
        t2 = 1 if t2 > 1 else t2
        t2 = -1 if t2 < -1 else t2
        y = asin(t2)
        # y axis rotation angle
        t3 = 2 * (self.w * self.z + self.x * self.y)
        t4 = 1 - 2 * (self.y * self.y + self.z * self.z)
        z = atan2(t3, t4)
        return degrees(x), degrees(y), degrees(z)

    def conjugate(self) -> 'Quaternion':
        """Quaternion conjugation

        Returns:
            Quaternion: Conjugated quaternion (x, y, z, w)

        >>> Quaternion.from_euler(90, -90, 90, 12).conjugate().to_euler()
        180.0, -90.0, 180.0

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # Calculation
        return Quaternion(
            x = -self.x, 
            y = -self.y, 
            z = -self.z, 
            w = self.w
        )

    def multiply_by(self, quaternion: 'Quaternion',  precision: int) -> 'Quaternion':
        """Quaternion multiplication

        Returns:
            Quaternion: Multiplication product quaternion (x, y, z, w)

        >>> Quaternion.identity().multiply_by(Quaternion.from_euler(90, 0, 0, 12), 12)
        0.707106781187, 0.0, 0.0, 0.707106781187

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # Calculation
        return Quaternion(
            x = float(round(self.w * quaternion.x + self.x * quaternion.w + self.y * quaternion.z - self.z * quaternion.y, precision)),
            y = float(round(self.w * quaternion.y + self.y * quaternion.w + self.z * quaternion.x - self.x * quaternion.z, precision)),
            z = float(round(self.w * quaternion.z + self.z * quaternion.w + self.x * quaternion.y - self.y * quaternion.x, precision)),
            w = float(round(self.w * quaternion.w - self.x * quaternion.x - self.y * quaternion.y - self.z * quaternion.z, precision))
        )

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z), 
                str(self.w)
            )
        )

romp_test.py:

#!/usr/bin/env python3

from log import Log
import sys
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion
import numpy as np
import romp, cv2
from scipy.spatial.transform import Rotation as r

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
subject_id = 0
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_root_position = outputs["cam_trans"][subject_id]
smpl_rotations_by_axis = outputs["smpl_thetas"][subject_id].reshape(24, 3)

# Info
rotation_by_axis = smpl_rotations_by_axis[0] # Pelvis
rotation_by_quaternion = r.from_rotvec(rotation_by_axis).as_quat()
print("ROMP settings: " + str(settings))
print("Root position: " + str(smpl_root_position))
print("Pelvis rotation by axis: " + str(rotation_by_axis))
print("Pelvis rotation by quaternion: " + str(rotation_by_quaternion))

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    13 : "LeftShoulder", # L_Collar
    14 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    16 : "LeftUpperArm", # L_Shoulder
    17 : "RightUpperArm", # R_Shoulder
    18 : "LeftLowerArm", # L_Elbow
    19 : "RightLowerArm", # R_Elbow
    20 : "LeftHand", # L_Wrist
    21 : "RightHand", # R_Wrist
    22 : "LeftMiddleProximal", # L_Hand
    23 : "RightMiddleProximal" # R_hand
}

vrm_swapped_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    14 : "LeftShoulder", # L_Collar
    13 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    17 : "LeftUpperArm", # L_Shoulder
    16 : "RightUpperArm", # R_Shoulder
    19 : "LeftLowerArm", # L_Elbow
    18 : "RightLowerArm", # R_Elbow
    21 : "LeftHand", # L_Wrist
    20 : "RightHand", # R_Wrist
    23 : "LeftMiddleProximal", # L_Hand
    22 : "RightMiddleProximal" # R_hand
}

#vrm_bone_names = vrm_swapped_bone_names

# VMC
vmc = VMCAssistant(connection['host'], connection['port'], connection['name'])
bones = []
for index, rotation in enumerate(smpl_rotations_by_axis):
    bone_name = vrm_bone_names[index]
    rotation = r.from_rotvec(rotation).as_quat()
    rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])
    """
    if bone_name in ("LeftShoulder", "LeftUpperArm", "LeftLowerArm", "LeftHand", "LeftMiddleProximal",
                     "RightShoulder", "RightUpperArm", "RightLowerArm", "RightHand", "RightMiddleProximal"):
        rotation = rotation.conjugate()
    """
    bones.append(
        [
            Bone(bone_name),
            Position(0.0, 0.0, 0.0),
            rotation
        ]
    )

bones += [
    [Bone("LeftEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("Jaw"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("LeftMiddleProximal"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("RightMiddleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)]
]
vmc.send_root_transform(
    Position(smpl_root_position[0], smpl_root_position[1], smpl_root_position[2]),
    bones[0][2].multiply_by(Quaternion.from_euler(0, 90, 0, 12), 12) # Hips / Pelvis rotation
)
vmc.send_bones_transform(bones)
vmc.send_available_states(1)
vmc.send_relative_time()

[Arthur151]

Hi, Vivian,

Happy to see the progress.

I can see that you intend to fix the retargeting problem via swapping / flipping the 3D rotation. But I don't think the problem could be well fixed in this way.

About the swapping: We can see that the mapping for leg joints is right. So it is not likely that the left/right arm joints are wrong.

I think the problem is still about the difference in the orientation of 3D coordinates for arm joints between SMPL and avatar.

Besides, for more reasonable testing the effectiveness of code, maybe we can animate the avatar with motion sequence to see if it is right. In that way, the orientation between different 3D coordinates would be very obvious.

[vivi90]

@Arthur151

I think the problem is still about the difference in the orientation of 3D coordinates for arm joints between SMPL and avatar.

So this time i tested it without additional rotations:

#!/usr/bin/env python3

from log import Log
import sys
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion
import numpy as np
import romp, cv2
from scipy.spatial.transform import Rotation as r

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
subject_id = 0
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_root_position = outputs["cam_trans"][subject_id]
smpl_rotations_by_axis = outputs["smpl_thetas"][subject_id].reshape(24, 3)

# Info
rotation_by_axis = smpl_rotations_by_axis[0] # Pelvis
rotation_by_quaternion = r.from_rotvec(rotation_by_axis).as_quat()
print("ROMP settings: " + str(settings))
print("Root position: " + str(smpl_root_position))
print("Pelvis rotation by axis: " + str(rotation_by_axis))
print("Pelvis rotation by quaternion: " + str(rotation_by_quaternion))

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    13 : "LeftShoulder", # L_Collar
    14 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    16 : "LeftUpperArm", # L_Shoulder
    17 : "RightUpperArm", # R_Shoulder
    18 : "LeftLowerArm", # L_Elbow
    19 : "RightLowerArm", # R_Elbow
    20 : "LeftHand", # L_Wrist
    21 : "RightHand", # R_Wrist
    22 : "LeftMiddleProximal", # L_Hand
    23 : "RightMiddleProximal" # R_hand
}

vrm_swapped_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    14 : "LeftShoulder", # L_Collar
    13 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    17 : "LeftUpperArm", # L_Shoulder
    16 : "RightUpperArm", # R_Shoulder
    19 : "LeftLowerArm", # L_Elbow
    18 : "RightLowerArm", # R_Elbow
    21 : "LeftHand", # L_Wrist
    20 : "RightHand", # R_Wrist
    23 : "LeftMiddleProximal", # L_Hand
    22 : "RightMiddleProximal" # R_hand
}

#vrm_bone_names = vrm_swapped_bone_names

# VMC
vmc = VMCAssistant(connection['host'], connection['port'], connection['name'])
bones = []
for index, rotation in enumerate(smpl_rotations_by_axis):
    bone_name = vrm_bone_names[index]
    rotation = r.from_rotvec(rotation).as_quat()
    rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])
    """
    if bone_name in ("LeftShoulder", "LeftUpperArm", "LeftLowerArm", "LeftHand", "LeftMiddleProximal",
                     "RightShoulder", "RightUpperArm", "RightLowerArm", "RightHand", "RightMiddleProximal"):
        rotation = rotation.conjugate()
    """
    bones.append(
        [
            Bone(bone_name),
            Position(0.0, 0.0, 0.0),
            rotation
        ]
    )

bones += [
    [Bone("LeftEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("Jaw"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("LeftMiddleProximal"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("RightMiddleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)]
]
vmc.send_root_transform(
    Position(smpl_root_position[0], smpl_root_position[1], smpl_root_position[2]),
    bones[0][2] # .multiply_by(Quaternion.from_euler(0, 90, 0, 12), 12) # Hips / Pelvis rotation
)
vmc.send_bones_transform(bones)
vmc.send_available_states(1)
vmc.send_relative_time()

And get the following values (captured by Protokol):

CONNECT    | ENDPOINT([::]:39539)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Root/Pos) STRING(root) FLOAT(-0.017495353) FLOAT(0.3001184) FLOAT(1.3419644) FLOAT(-0.7916114) FLOAT(0.092278905) FLOAT(-0.5999143) FLOAT(0.07027662)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Hips) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.7916114) FLOAT(0.092278905) FLOAT(-0.5999143) FLOAT(0.07027662)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftUpperLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.569858) FLOAT(-0.06745373) FLOAT(0.21782088) FLOAT(0.7894719)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightUpperLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.09012153) FLOAT(-0.08178122) FLOAT(-0.20650071) FLOAT(0.9708488)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Spine) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.15954086) FLOAT(0.056978386) FLOAT(-0.01746822) FLOAT(0.9853908)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLowerLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.51568484) FLOAT(0.071990184) FLOAT(-0.080458954) FLOAT(0.84994876)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLowerLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.64322734) FLOAT(0.124634884) FLOAT(-0.03805522) FLOAT(0.75450414)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Chest) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.009237677) FLOAT(-0.00467792) FLOAT(-0.015887119) FLOAT(0.9998202)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftFoot) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.014397013) FLOAT(0.055690825) FLOAT(-0.11832434) FLOAT(0.9913075)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightFoot) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.10581176) FLOAT(-0.046522453) FLOAT(0.08669153) FLOAT(0.989507)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(UpperChest) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.014591641) FLOAT(0.038391713) FLOAT(-0.0090085715) FLOAT(0.9991156)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftToes) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.14389484) FLOAT(0.09746717) FLOAT(0.2301151) FLOAT(0.9575184)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightToes) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.17262207) FLOAT(-0.05242086) FLOAT(-0.11825366) FLOAT(0.9764578)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Neck) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.08807893) FLOAT(-0.010544569) FLOAT(-0.009392015) FLOAT(0.9960134)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftShoulder) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.073637694) FLOAT(-0.14232136) FLOAT(-0.026770841) FLOAT(0.9867145)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightShoulder) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.062228765) FLOAT(0.19211994) FLOAT(0.015868997) FLOAT(0.97926795)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Head) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.03832356) FLOAT(-0.024623983) FLOAT(-0.002712541) FLOAT(0.9989583)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftUpperArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.05442533) FLOAT(-0.32393435) FLOAT(-0.309805) FLOAT(0.8922585)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightUpperArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.10145097) FLOAT(0.30820185) FLOAT(0.25927484) FLOAT(0.909668)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLowerArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.007941601) FLOAT(-0.78527176) FLOAT(0.33539173) FLOAT(0.5203821)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLowerArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.0941422) FLOAT(0.67905384) FLOAT(-0.2797612) FLOAT(0.67212856)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftHand) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.07618274) FLOAT(-0.07240124) FLOAT(0.11218269) FLOAT(0.988114)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightHand) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.0491007) FLOAT(0.05441978) FLOAT(-0.09431752) FLOAT(0.9928403)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftEye) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightEye) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Jaw) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/OK) INT32(1)
RECEIVE    | ENDPOINT([::1]:50091) ADDRESS(/VMC/Ext/T) FLOAT(1.6553057e+9)
DISCONNECT | ENDPOINT([::]:39539)

Then i applied just the following rotations by hand inside Blender:

'RightShoulder', x=-0.062228765, y=0.19211994, z=0.015868997, w=0.97926795
'RightUpperArm', x=-0.10145097, y=0.30820185, z=0.25927484, w=0.909668
'RightLowerArm', x=-0.0941422, y=0.67905384, z=-0.2797612, w=0.67212856
'RightHand', x=-0.0491007, y=0.05441978, z=-0.09431752, w=0.9928403

And get this result: Why this different behaviour?

Besides, for more reasonable testing the effectiveness of code, maybe we can animate the avatar with motion sequence to see if it is right. In that way, the orientation between different 3D coordinates would be very obvious.

Maybe. Maybe not. I don't know.

Sorry, but at this point i need to give up my project, because i am running out of my deadline.

[vivi90]

This way it looks in Blender with an SMPL model:

[vivi90]

@Arthur151 Okay i tested some values in Blender to hopefully get an idea whats wrong: So i calculated some rotations with my script:

# Detected rotations rotations
'RightShoulder', x=-0.062228765, y=0.19211994, z=0.015868997, w=0.97926795
'RightUpperArm', x=-0.10145097, y=0.30820185, z=0.25927484, w=0.909668
'RightLowerArm', x=-0.0941422, y=0.67905384, z=-0.2797612, w=0.67212856
'RightHand', x=-0.0491007, y=0.05441978, z=-0.09431752, w=0.9928403

# Detected rotations + 90 degree (rotation = rotation.multiply_by(Quaternion.from_euler(0, 90, 0, 12), 12))
'RightShoulder', x=-0.055223458, y=0.8282963, z=-0.032781307, w=0.55659765
'RightUpperArm', x=-0.25507167, y=0.86116403, z=0.11159832, w=0.4253008
'RightLowerArm', x=0.13125245, y=0.9554302, z=-0.26438963, w=-0.0048969327
'RightHand', x=0.03197312, y=0.7405247, z=-0.101412, w=0.6635635

# Detected rotations - 90 degree (rotation = rotation.multiply_by(Quaternion.from_euler(0, -90, 0, 12), 12))
'RightShoulder', x=-0.032781307, y=-0.55659765, z=0.055223458, w=0.8282963
'RightUpperArm', x=0.11159832, y=-0.4253008, z=0.25507167, w=0.86116403
'RightLowerArm', x=-0.26438963, y=0.0048969327, z=-0.13125245, w=0.9554302
'RightHand', x=-0.101412, y=-0.6635635, z=-0.03197312, w=0.7405247

After that i tested it with my model: and with the SMPL model: Everytime the result seems weird and unlogic so i tested another idea. Using the SMPL model in Blender with the detected rotations (in quaternion) by my script, but WITHOUT some additional rotation. Then i switched in the 'Bone Properties' the 'Mode' to 'XYZ Euler' and compared the calculated results by Blender with my own (by my script):

# Without additional rotation by my script
'RightLowerArm', x=-0.0941422, y=0.67905384, z=-0.2797612, w=0.67212856

# Would be (calculated by Blender)
'X': -83.239d
'Y': 59.3332d
'Z': -98.8831d

# y-90 in Blender (correct)
'X': -83.239d
'Y': -30.6668d
'Z': -98.8831d

# In quaternion calculated by Blender
'RightLowerArm', x=-0.566682, y=0.358106, z=-0.661946, w=0.33535

# Wrong calculated by my script
'RightLowerArm', x=-0.26438963, y=0.0048969327, z=-0.13125245, w=0.9554302

# Would be (wrong)
'X': -30.4917d
'Y': -3.44247d
'Z': -14.7056d

So it seems my Quaternion calculations are wrong! 🤣 (WRONG) Quaternion class in vmc.py:

class Quaternion:
    def __init__(self, x: float, y: float, z: float, w: float) -> None:
        if round(pow(x, 2) + pow(y, 2) + pow(z, 2) + pow(w, 2), 1) == 1:
            self.x = float(x)
            self.y = float(y)
            self.z = float(z)
            self.w = float(w)
        else:
            raise ValueError(
                "Invalid quaternion values: x={}, y={}, z={}, w={}".format(
                    x,
                    y,
                    z,
                    w
                )
            )

    @classmethod
    def identity(cls) -> 'Quaternion':
        """Creates an identity quaternion and returns it

        Returns:
            Quaternion: Created identity quaternion (x, y, z, w)

        >>> Quaternion.identity()
        0.0, 0.0, 0.0, 1.0

        """
        return cls(
            x = float(0.0),
            y = float(0.0),
            z = float(0.0),
            w = float(1.0)
        )

    @classmethod
    def from_euler(cls, 
                   phi: float, theta: float, psi: float,
                   precision: int) -> 'Quaternion':
        """Creates an quaternion by euler angles and returns it

        Args:
            phi (float): Rotation angle around the X axis
            theta (float): Rotation angle around the Y axis
            psi (float): Rotation angle around the Y axis
            precision (int): Round the results to 'precision' digits after the decimal point

        Returns:
            Quaternion: Created quaternion (x, y, z, w)

        >>> Quaternion.from_euler(-90, -180, 90, 12)
        0.5, -0.5, -0.5, 0.5

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        cos_phi_half = cos(radians(phi) / 2)
        sin_phi_half = sin(radians(phi) / 2)
        # y axis rotation angle
        cos_theta_half = cos(radians(theta) / 2)
        sin_theta_half = sin(radians(theta) / 2)
        # z axis rotation angle
        cos_psi_half = cos(radians(psi) / 2)
        sin_psi_half = sin(radians(psi) / 2)
        # Calculation
        return cls(
            x = float(round(sin_phi_half * cos_theta_half * cos_psi_half - cos_phi_half * sin_theta_half * sin_psi_half, precision)),
            y = float(round(cos_phi_half * sin_theta_half * cos_psi_half + sin_phi_half * cos_theta_half * sin_psi_half, precision)),
            z = float(round(cos_phi_half * cos_theta_half * sin_psi_half - sin_phi_half * sin_theta_half * cos_psi_half, precision)),
            w = float(round(cos_phi_half * cos_theta_half * cos_psi_half + sin_phi_half * sin_theta_half * sin_psi_half, precision))
        )

    def to_euler(self) -> tuple[float, float, float]:
        """Exports an quaternion as an euler angle tuple

        Returns:
            tuple[float, float, float]: (x, y, z)

        >>> Quaternion.from_euler(-90, -180, 90, 12).to_euler()
        90.0, 0.0, -90.0

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # x axis rotation angle
        t0 = 2 * (self.w * self.x + self.y * self.z)
        t1 = 1 - 2 * (self.x * self.x + self.y * self.y)
        x = atan2(t0, t1)
        # y axis rotation angle
        t2 = 2 * (self.w * self.y - self.z * self.x)
        t2 = 1 if t2 > 1 else t2
        t2 = -1 if t2 < -1 else t2
        y = asin(t2)
        # y axis rotation angle
        t3 = 2 * (self.w * self.z + self.x * self.y)
        t4 = 1 - 2 * (self.y * self.y + self.z * self.z)
        z = atan2(t3, t4)
        return degrees(x), degrees(y), degrees(z)

    def conjugate(self) -> 'Quaternion':
        """Quaternion conjugation

        Returns:
            Quaternion: Conjugated quaternion (x, y, z, w)

        >>> Quaternion.from_euler(90, -90, 90, 12).conjugate().to_euler()
        180.0, -90.0, 180.0

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # Calculation
        return Quaternion(
            x = -self.x, 
            y = -self.y, 
            z = -self.z, 
            w = self.w
        )

    def multiply_by(self, quaternion: 'Quaternion',  precision: int) -> 'Quaternion':
        """Quaternion multiplication

        Returns:
            Quaternion: Multiplication product quaternion (x, y, z, w)

        >>> Quaternion.identity().multiply_by(Quaternion.from_euler(90, 0, 0, 12), 12)
        0.707106781187, 0.0, 0.0, 0.707106781187

        .. _Source:
            https://www.meccanismocomplesso.org/en/hamiltons-quaternions-and-3d-rotation-with-python

        """
        # Calculation
        return Quaternion(
            x = float(round(self.w * quaternion.x + self.x * quaternion.w + self.y * quaternion.z - self.z * quaternion.y, precision)),
            y = float(round(self.w * quaternion.y + self.y * quaternion.w + self.z * quaternion.x - self.x * quaternion.z, precision)),
            z = float(round(self.w * quaternion.z + self.z * quaternion.w + self.x * quaternion.y - self.y * quaternion.x, precision)),
            w = float(round(self.w * quaternion.w - self.x * quaternion.x - self.y * quaternion.y - self.z * quaternion.z, precision))
        )

    def __str__(self) -> str:
        return ", ".join(
            (
                str(self.x), 
                str(self.y), 
                str(self.z), 
                str(self.w)
            )
        )

[vivi90]

@Arthur151 Sorry that i ask this stupid thing but how to rotate an rotation by 90 degree around the y axis with scipy? This doesn't work:

#!/usr/bin/env python3

#from math import radians, degrees, cos, sin, atan2, asin, pow, floor
#import numpy as np
from scipy.spatial.transform import Rotation

#r = Rotation.from_rotvec(rotation_by_axis).as_quat()
r = Rotation.from_quat([-0.0941422, 0.67905384, -0.2797612, 0.67212856]) # example
print(r.as_euler('xyz', degrees=True))
print(r.apply([0, 90, 0]))

Wrong:

[-83.23902624  59.33323676 -98.88314731]
[ 22.33941658  74.31676511 -45.58474405]

[vivi90]

After asking here i still have at least an bad way of rotating:

if bone_name in (#"LeftShoulder", "LeftUpperArm", "LeftLowerArm", "LeftHand", "LeftMiddleProximal",
                  "RightLowerArm"): #"RightShoulder", "RightUpperArm", "RightLowerArm", "RightHand", "RightMiddleProximal"):
    rotation = r.from_quat([rotation.x, rotation.y, rotation.z, rotation.w])
    rotation = rotation.as_euler('xyz', degrees=True)
    print(rotation)
    rotation = r.from_euler('xyz', [rotation[0], rotation[1]-90, rotation[2]], degrees=True)
    print(rotation.as_euler('xyz', degrees=True))
    rotation = rotation.as_quat()
    rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])
    #rotation = rotation.multiply_by(Quaternion.from_euler(0, -90, 0, 12), 12)

leads to:

[-83.23903078  59.33323752 -98.88315157]
[-83.23903078 -30.66676248 -98.88315157]

But it still doesn't work: Because sometimes my calculations:

if bone_name in ("LeftShoulder", "LeftUpperArm", "LeftLowerArm", "LeftHand", "LeftMiddleProximal",
                  "RightShoulder", "RightUpperArm", "RightLowerArm", "RightHand", "RightMiddleProximal"):
    rotation = r.from_quat([rotation.x, rotation.y, rotation.z, rotation.w])
    rotation = rotation.as_euler('xyz', degrees=True)
    print(rotation)
    rotation = r.from_euler('xyz', [rotation[0], rotation[1]-90, rotation[2]], degrees=True)
    print(rotation.as_euler('xyz', degrees=True))
    rotation = rotation.as_quat()
    rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])

are not working:

[  -8.25897711  -16.54712028    -1.90525288]
[ 171.74102289  -73.45287972   178.09474712]

[  -7.18492129   22.22525264     0.44373851]
[  -7.18492129  -67.77474736     0.44373851]

[   7.52491766  -37.71896037   -40.86915413]
[-172.47508234  -52.28103963   139.13084587]

[  -1.79610826   37.83068221    31.20184248]
[  -1.79610826  -52.16931779    31.20184248]

[-113.5719734   -54.28744892   141.73007557]
[  66.4280266   -35.71255108   -38.26992443]

[ -83.23903078   59.33323752   -98.88315157]
[ -83.23903078  -30.66676248   -98.88315157]

[  -9.67960912   -7.23784945    13.56800885]
[ 170.32039088  -82.76215055  -166.43199115]

[  -6.21695895    5.66996884   -11.16152822]
[  -6.21695895  -84.33003116   -11.16152822]

[   0.            0.             0.        ]
[   0.          -90.             0.        ]

[   0.            0.             0.        ]
[   0.          -90.             0.        ]

Log:

CONNECT    | ENDPOINT([::]:39539)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Root/Pos) STRING(root) FLOAT(-0.017495353) FLOAT(0.3001184) FLOAT(1.3419644) FLOAT(-0.7916114) FLOAT(0.092278905) FLOAT(-0.5999143) FLOAT(0.07027662)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Hips) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.7916114) FLOAT(0.092278905) FLOAT(-0.5999143) FLOAT(0.07027662)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftUpperLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.569858) FLOAT(-0.06745373) FLOAT(0.21782088) FLOAT(0.7894719)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightUpperLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.09012153) FLOAT(-0.08178122) FLOAT(-0.20650071) FLOAT(0.9708488)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Spine) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.15954086) FLOAT(0.056978386) FLOAT(-0.01746822) FLOAT(0.9853908)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLowerLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.51568484) FLOAT(0.071990184) FLOAT(-0.080458954) FLOAT(0.84994876)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLowerLeg) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.64322734) FLOAT(0.124634884) FLOAT(-0.03805522) FLOAT(0.75450414)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Chest) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.009237677) FLOAT(-0.00467792) FLOAT(-0.015887119) FLOAT(0.9998202)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftFoot) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.014397013) FLOAT(0.055690825) FLOAT(-0.11832434) FLOAT(0.9913075)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightFoot) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.10581176) FLOAT(-0.046522453) FLOAT(0.08669153) FLOAT(0.989507)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(UpperChest) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.014591641) FLOAT(0.038391713) FLOAT(-0.0090085715) FLOAT(0.9991156)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftToes) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.14389484) FLOAT(0.09746717) FLOAT(0.2301151) FLOAT(0.9575184)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightToes) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.17262207) FLOAT(-0.05242086) FLOAT(-0.11825366) FLOAT(0.9764578)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Neck) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.08807893) FLOAT(-0.010544569) FLOAT(-0.009392015) FLOAT(0.9960134)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftShoulder) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.056347046) FLOAT(-0.7985925) FLOAT(-0.067624934) FLOAT(0.5954006)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightShoulder) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.04986038) FLOAT(-0.5566638) FLOAT(-0.031727884) FLOAT(0.828633)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Head) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.03832356) FLOAT(-0.024623983) FLOAT(-0.002712541) FLOAT(0.9989583)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftUpperArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.28565973) FLOAT(-0.8495049) FLOAT(-0.098288245) FLOAT(0.4325267)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightUpperArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.10467765) FLOAT(-0.4272332) FLOAT(0.23487566) FLOAT(0.8668032)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLowerArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.4084832) FLOAT(-0.41326824) FLOAT(-0.10234998) FLOAT(0.80738795)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLowerArm) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.56668204) FLOAT(0.35810596) FLOAT(-0.6619457) FLOAT(0.33534974)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftHand) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0.032934412) FLOAT(-0.7490079) FLOAT(0.014948777) FLOAT(0.6615732)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightHand) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.10518698) FLOAT(-0.66319466) FLOAT(-0.10820419) FLOAT(0.733076)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.70710677) FLOAT(0) FLOAT(0.70710677)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(-0.70710677) FLOAT(0) FLOAT(0.70710677)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftEye) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightEye) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(Jaw) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftThumbDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftIndexDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftMiddleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftRingDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(LeftLittleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightThumbDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightIndexDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightMiddleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightRingDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleProximal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleIntermediate) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/Bone/Pos) STRING(RightLittleDistal) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(0) FLOAT(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/OK) INT32(1)
RECEIVE    | ENDPOINT([::1]:54510) ADDRESS(/VMC/Ext/T) FLOAT(1.655327e+9)
DISCONNECT | ENDPOINT([::]:39539)

romp_test.py:

#!/usr/bin/env python3

from log import Log
import sys
from vmc import Assistant as VMCAssistant, Bone, Position, Quaternion
import numpy as np
import romp, cv2
from scipy.spatial.transform import Rotation as r

# Configuration
connection = {
    "host" : "localhost",
    "port" : 39539,
    "name" : "example"
}

# Logging
sys.stdout = Log(filename = "vmc.log", is_error = False)
sys.stderr = Log(filename = "vmc.log", is_error = True)

# ROMP
settings = romp.main.default_settings
subject_id = 0
romp_model = romp.ROMP(settings)
outputs = romp_model(cv2.imread('pose.png'))
smpl_root_position = outputs["cam_trans"][subject_id]
smpl_rotations_by_axis = outputs["smpl_thetas"][subject_id].reshape(24, 3)

# Info
rotation_by_axis = smpl_rotations_by_axis[0] # Pelvis
rotation_by_quaternion = r.from_rotvec(rotation_by_axis).as_quat()
print("ROMP settings: " + str(settings))
print("Root position: " + str(smpl_root_position))
print("Pelvis rotation by axis: " + str(rotation_by_axis))
print("Pelvis rotation by quaternion: " + str(rotation_by_quaternion))

# VRM
# (Just the logical bone names. Not the actual ones [nodes] in the rig)
vrm_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    13 : "LeftShoulder", # L_Collar
    14 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    16 : "LeftUpperArm", # L_Shoulder
    17 : "RightUpperArm", # R_Shoulder
    18 : "LeftLowerArm", # L_Elbow
    19 : "RightLowerArm", # R_Elbow
    20 : "LeftHand", # L_Wrist
    21 : "RightHand", # R_Wrist
    22 : "LeftMiddleProximal", # L_Hand
    23 : "RightMiddleProximal" # R_hand
}

vrm_swapped_bone_names = {
    0 : "Hips", # Pelvis
    1 : "LeftUpperLeg", # L_Hip
    2 : "RightUpperLeg", # R_Hip
    3 : "Spine", # Spine1
    4 : "LeftLowerLeg", # L_Knee
    5 : "RightLowerLeg", # R_Knee
    6 : "Chest", # Spine2
    7 : "LeftFoot", # L_Ankle
    8 : "RightFoot", # R_Ankle
    9 : "UpperChest", # Spine3
    10 : "LeftToes", # L_Foot
    11 : "RightToes", # R_Foot
    12 : "Neck", # Neck
    14 : "LeftShoulder", # L_Collar
    13 : "RightShoulder", # R_Collar
    15 : "Head", # Head
    17 : "LeftUpperArm", # L_Shoulder
    16 : "RightUpperArm", # R_Shoulder
    19 : "LeftLowerArm", # L_Elbow
    18 : "RightLowerArm", # R_Elbow
    21 : "LeftHand", # L_Wrist
    20 : "RightHand", # R_Wrist
    23 : "LeftMiddleProximal", # L_Hand
    22 : "RightMiddleProximal" # R_hand
}

#vrm_bone_names = vrm_swapped_bone_names

# VMC
vmc = VMCAssistant(connection['host'], connection['port'], connection['name'])
bones = []
for index, rotation in enumerate(smpl_rotations_by_axis):
    bone_name = vrm_bone_names[index]
    rotation = r.from_rotvec(rotation).as_quat()
    rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])
    if bone_name in ("LeftShoulder", "LeftUpperArm", "LeftLowerArm", "LeftHand", "LeftMiddleProximal",
                     "RightShoulder", "RightUpperArm", "RightLowerArm", "RightHand", "RightMiddleProximal"):
        rotation = r.from_quat([rotation.x, rotation.y, rotation.z, rotation.w])
        rotation = rotation.as_euler('xyz', degrees=True)
        print(rotation)
        rotation = r.from_euler('xyz', [rotation[0], rotation[1]-90, rotation[2]], degrees=True)
        print(rotation.as_euler('xyz', degrees=True))
        rotation = rotation.as_quat()
        rotation = Quaternion(rotation[0], rotation[1], rotation[2],  rotation[3])
        #rotation = rotation.multiply_by(Quaternion.from_euler(0, -90, 0, 12), 12)
    bones.append(
        [
            Bone(bone_name),
            Position(0.0, 0.0, 0.0),
            rotation
        ]
    )

bones += [
    [Bone("LeftEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightEye"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("Jaw"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("LeftMiddleProximal"), Position(0.0, 0.0, 0.0),  Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("LeftLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightThumbDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightIndexDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
  # [Bone("RightMiddleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightMiddleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightRingDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleProximal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleIntermediate"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)],
    [Bone("RightLittleDistal"), Position(0.0, 0.0, 0.0), Quaternion(0.0, 0.0, 0.0, 1.0)]
]
vmc.send_root_transform(
    Position(smpl_root_position[0], smpl_root_position[1], smpl_root_position[2]),
    bones[0][2] # .multiply_by(Quaternion.from_euler(0, 90, 0, 12), 12) # Hips / Pelvis rotation
)
vmc.send_bones_transform(bones)
vmc.send_available_states(1)
vmc.send_relative_time()

[vivi90]

Sorry, i give up. So draft public now: https://github.com/vivi90/python-vmc

[vivi90]

Closed due to possible deadline (30th of June 2022) exceeding: https://github.com/vivi90/python-vmc/issues/1

[vivi90]

@Arthur151 Project reopened due to deadline extension until 1st of August 2022. So i will fix at first the broken communication timestamps and after that continue on fixing the calculation bug.

[Arthur151]

Welcome back~

[vivi90]

@Arthur151 Timestamps fixed and so VMC protocol communication stability improved: https://github.com/emilianavt/VSeeFaceReleases/issues/15#issuecomment-1164711035

[vivi90]

@Arthur151 The bone rotations, i get from ROMP: Are these local (relative to the parent bone) rotations or global ('world' related) rotations?

[Arthur151]

@vivi90 All SMPL joints rotation estimated by ROMP is relative (local) rotation here.

[vivi90]

@Arthur151 Another question: These joints are SMPL optimized. Is there any way to get out someway raw joints (not for SMPL optimized)?

[Arthur151]

I didn't get it. What do you mean by SMPL optimized or raw joints.

[vivi90]

[..] What do you mean by SMPL optimized or raw joints.

I mean:

• SMPL optimized: Joint data optimized for the SMPL skeleton (especially for some of the 90 degrees rotatet bones).
• Raw: Not optimized for SMPL skeleton. More generic.