bvh2smpl

[dongran]

Thank you for providing this great tool!

I was wondering if there is a way to convert exported BVH back to SMPL. The npz (only trans and poses) for SMPL is enough since there is no loss and scaling inside the original BVH file.

I have converted my own data set to the BVH skeleton you adopted in this project. However, I could find a way to back quaternion to axis-angle because you divide axis-angle by its norm for normalization.

Could you give me some pieces of advice for bvh2smpl?

Thank you!

[KosukeFukazawa]

Thank you for your interest in this repository!

The conversion from quaternion to axis-angle may be helpful for codes such as Pytorch3D. (https://pytorch3d.readthedocs.io/en/latest/modules/transforms.html#pytorch3d.transforms.quaternion_to_axis_angle)

Note that the order of joints in BVH is different from SMPL due to the hierarchical structure. SMPL: {0: "pelvis", 1: "left_hip", 2: "right_hip", ... } BVH. : {0: "pelvis", 1: "left_hip", 2: "Left_knee", ... } You can check SMPL order from here https://github.com/vchoutas/smplx/blob/main/smplx/joint_names.py

If you have any problems, please ask me again.

[dongran]

Thank you very much for your quick reply!

I found a way to change BVH Euler angles to axis-angle. However, there are still some problems here.

I use gBR_sBM_cAll_d04_mBR0_ch01.bvh as an example to convert the root joint to axis-angle. When I compared the converted axis-angle with it in the root joint read from the PKL file, I found too many numerical differences between the two data.

I attached the differences as a figure. Did I miss something after I obtained the converted axis-angle?

I would appreciate it If you could give some comments on this issue.

Also thank you for your advice about the joint difference! I will pay attention to mapping them.

[KosukeFukazawa]

Hi dongran,

As far as I can guess, the following are possible causes:

• You need to convert bvh rotations to radian like np.radian(rotations), because bvh files are written in degree format.
• You need to check euler order(default to "zyx").

You can check quaternion to axis-angle function from my another repo(it supports AMASS). https://github.com/KosukeFukazawa/CharacterAnimationTools/blob/c9d366e4b61dc020634b26553faf1cdb00116bb1/util/quat.py#L147

[dongran]

Hi KosukeFukazawa,

Thank you for your kind advice!

It seems I used the wrong package to convert quaternion to axis-angle. After I tried the method you provided to me, I can correctly calculate SMPL -> BVH ->SMPL -> BVH using this smpl2bvh repo! I also checked the BVH files. They have totally the same poses. Thank you so much!

By the way, do you also provide a visualizer for the SMPL sequence in your other repos? Just like GIF files in this repo, directly showing the SMPL sequence to users.

[KosukeFukazawa]

That's good!

By the way, do you also provide a visualizer for the SMPL sequence in your other repos? Just like GIF files in this repo, directly showing the SMPL sequence to users.

I'm so sorry. I couldn't find my visualize script. I think I wrote it using matplotlib...

[dongran]

Thank you for your reply!

I will try it by myself using matplotlib.

Thank you for your kind help! Your repos are really helpful for our project.

Thank you so much!

[KosukeFukazawa]

I'm glad I could help your project! Good luck!!

[limshoonkit]

[RESOLVED] Hi there, is it possible to provide me with the code to convert bvh to smpl in aist++ dataset format?

dumping code here, the euler rotation order is important depending on the .bvh file

import sys
import numpy as np
from bvh import Bvh
from scipy.spatial.transform import Rotation as R
import math

JOINT_MAP = {
    # 'BVH joint name': 'SMPLX joint index'
    'hips_JNT': 0,
    'l_upleg_JNT': 1,
    'r_upleg_JNT': 2,
    'spine_JNT': 3,
    'l_leg_JNT': 4,
    'r_leg_JNT': 5,
    'spine1_JNT': 6,
    'l_foot_JNT': 7,
    'r_foot_JNT': 8,
    'spine2_JNT': 9,
    'l_toebase_JNT': 10,
    'r_toebase_JNT': 11,
    'neck_JNT': 12,
    'l_shoulder_JNT': 13,
    'r_shoulder_JNT': 14,
    'head_JNT': 15,
    'l_arm_JNT': 16,
    'r_arm_JNT': 17,
    'l_forearm_JNT': 18,
    'r_forearm_JNT': 19,
    'l_hand_JNT': 20,
    'r_hand_JNT': 21,
    'l_handIndex1_JNT': 25,
    'l_handIndex2_JNT': 26,
    'l_handIndex3_JNT': 27,
    'l_handMiddle1_JNT': 28,
    'l_handMiddle2_JNT': 29,
    'l_handMiddle3_JNT': 30,
    'l_handPinky1_JNT': 31,
    'l_handPinky2_JNT': 32,
    'l_handPinky3_JNT': 33,
    'l_handRing1_JNT': 34,
    'l_handRing2_JNT': 35,
    'l_handRing3_JNT': 36,
    'l_handThumb1_JNT': 37,
    'l_handThumb2_JNT': 38,
    'l_handThumb3_JNT': 39,
    'r_handIndex1_JNT': 40,
    'r_handIndex2_JNT': 41,
    'r_handIndex3_JNT': 42,
    'r_handMiddle1_JNT': 43,
    'r_handMiddle2_JNT': 44,
    'r_handMiddle3_JNT': 45,
    'r_handPinky1_JNT': 46,
    'r_handPinky2_JNT': 47,
    'r_handPinky3_JNT': 48,
    'r_handRing1_JNT': 49,
    'r_handRing2_JNT': 50,
    'r_handRing3_JNT': 51,
    'r_handThumb1_JNT': 52,
    'r_handThumb2_JNT': 53,
    'r_handThumb3_JNT': 54,
}

def bvh_to_smplx(bvh_file, n_frames=None):
    with open(bvh_file, 'r') as f:
        mocap = Bvh(f.read())

    if n_frames is None:
        num_frames = len(mocap.frames)
    else:
        num_frames = min(n_frames, len(mocap.frames))

    num_frames_downsampled = math.ceil(num_frames)

    smplx_poses = np.zeros((num_frames_downsampled, 165))
    smplx_trans = np.zeros((num_frames_downsampled, 3))

    bvh_joint_names = set(mocap.get_joints_names())
    # print(bvh_joint_names)

    rotation_correction = R.from_euler('XYZ', [0, 0, 0], degrees=True)

    for i in range(0, num_frames, 1):
        print('Processing frame {}/{}'.format(i, num_frames), end='\r')
        for joint_name, joint_index in JOINT_MAP.items():
            # print(joint_name, joint_index)

            rotation = R.from_euler('ZXY', mocap.frame_joint_channels(i, joint_name, ['Zrotation', 'Xrotation', 'Yrotation']), degrees=True)

            if joint_name == 'hips_JNT':
                # rotation = rotation * rotation_correction
                rotation = rotation_correction * rotation

            smplx_poses[i, 3 * joint_index:3 * (joint_index + 1)] = rotation.as_rotvec()

            if joint_name == 'hips_JNT':
                x, y, z = mocap.frame_joint_channels(i, joint_name, ['Xposition', 'Yposition', 'Zposition'])
                smplx_trans[i] = np.array([x, -z, y])

    # mirror
    # smplx_trans[:, 1] *= -1

    scale_factor = 0.009
    smplx_trans *= scale_factor

    return smplx_trans, smplx_poses

import pickle

def save_pkl(output_file, smplx_trans, smplx_poses, gender='female', model_type='smplx', frame_rate=30):
    data = {
        'smpl_trans': smplx_trans,
        'smpl_poses': smplx_poses,
    }
    with open(output_file, 'wb') as f:
        pickle.dump(data, f)

if __name__ == '__main__':
    bvh_file = sys.argv[1]
    output_file = sys.argv[2]

    smplx_trans, smplx_poses = bvh_to_smplx(bvh_file, n_frames=2000)

    with open(bvh_file, 'r') as f:
        mocap = Bvh(f.read())
        frame_rate = 1.0 / mocap.frame_time
    print('frame_rate: ', frame_rate)
    save_pkl(output_file,smplx_trans,smplx_poses,frame_rate)

[vadeli]

@dongran Hi. Could you please share your BVH to SMPL conversion code? I need it for my project but can't find any available code. It would be a huge help. If you'd prefer to share it privately, my email is vida.adeli@gmail.com.

[limshoonkit]

@vadeli

import sys
import numpy as np
from bvh import Bvh
from scipy.spatial.transform import Rotation as R
import math

JOINT_MAP = {
    # 'BVH joint name': 'SMPL joint index'
    'hips_JNT': 0,
    'l_upleg_JNT': 1,
    'r_upleg_JNT': 2,
    'spine_JNT': 3,
    'l_leg_JNT': 4,
    'r_leg_JNT': 5,
    'spine1_JNT': 6,
    'l_foot_JNT': 7,
    'r_foot_JNT': 8,
    'spine2_JNT': 9,
    'l_toebase_JNT': 10,
    'r_toebase_JNT': 11,
    'neck_JNT': 12,
    'l_shoulder_JNT': 13,
    'r_shoulder_JNT': 14,
    'head_JNT': 15,
    'l_arm_JNT': 16,
    'r_arm_JNT': 17,
    'l_forearm_JNT': 18,
    'r_forearm_JNT': 19,
    'l_hand_JNT': 20,
    'r_hand_JNT': 21,
    'l_handIndex1_JNT': 22,
    'r_handIndex1_JNT': 23,
}

def bvh_to_smpl(bvh_file, n_frames=None):
    with open(bvh_file, 'r') as f:
        mocap = Bvh(f.read())

    if n_frames is None:
        num_frames = len(mocap.frames)
    else:
        num_frames = min(n_frames, len(mocap.frames))

    num_frames_downsampled = math.ceil(num_frames)

    smpl_poses = np.zeros((num_frames_downsampled, 72))
    smpl_trans = np.zeros((num_frames_downsampled, 3))

    bvh_joint_names = set(mocap.get_joints_names())
    # print(bvh_joint_names)

    rotation_correction = R.from_euler('XYZ', [0, 0, 0], degrees=True)

    for i in range(0, num_frames, 1):
        print('Processing frame {}/{}'.format(i, num_frames), end='\r')
        for joint_name, joint_index in JOINT_MAP.items():
            # print(joint_name, joint_index)

            rotation = R.from_euler('ZXY', mocap.frame_joint_channels(i, joint_name, ['Zrotation', 'Xrotation', 'Yrotation']), degrees=True)

            if joint_name == 'hips_JNT':
                # rotation = rotation * rotation_correction
                rotation = rotation_correction * rotation

            smpl_poses[i, 3 * joint_index:3 * (joint_index + 1)] = rotation.as_rotvec()

            if joint_name == 'hips_JNT':
                x, y, z = mocap.frame_joint_channels(i, joint_name, ['Xposition', 'Yposition', 'Zposition'])
                smpl_trans[i] = np.array([x, -z, y])

    # mirror
    # smpl_trans[:, 1] *= -1

    scale_factor = 0.009
    smpl_trans *= scale_factor

    return smpl_trans, smpl_poses

import pickle

def save_pkl(output_file, smpl_trans, smpl_poses, gender='female', model_type='smpl', frame_rate=30, smpl_scaling=1.0):
    data = {
        'smpl_trans': smpl_trans.astype(np.float32),
        'smpl_poses': smpl_poses.astype(np.float32),
        'smpl_scaling' : np.array([smpl_scaling], dtype=np.float32), 
    }
    with open(output_file, 'wb') as f:
        pickle.dump(data, f)

if __name__ == '__main__':
    # python bvh2smpl.py data/input/1.bvh data/test/1.pkl
    bvh_file = sys.argv[1]
    output_file = sys.argv[2]

    smpl_trans, smpl_poses = bvh_to_smpl(bvh_file, n_frames=2000)

    with open(bvh_file, 'r') as f:
        mocap = Bvh(f.read())
        frame_rate = 1.0 / mocap.frame_time
    print('frame_rate: ', frame_rate)
    save_pkl(output_file,smpl_trans,smpl_poses,frame_rate)

[vadeli]

@limshoonkit Thank you so much for your quick reply. I used your code and tried to visualize the generated smpl mesh with obtained smpl_trans, smpl_poses. But I am getting rest pose for all the frames.

https://github.com/KosukeFukazawa/smpl2bvh/assets/16913712/903cf34c-c33e-4f88-b31f-1c551faa45f3

This is my code (my BVH file and its skeleton visualization is in this link):

import sys
import numpy as np
from bvh import Bvh
from scipy.spatial.transform import Rotation as R
import math
import torch
import pyrender
import imageio
import trimesh
import os
from tqdm import tqdm

import smplx
from smplx import SMPL

import trimesh.visual
os.environ['DISPLAY'] = ':1'

from smplx_visualization import euler_to_rotation_matrix, create_raymond_lights

WIDTH = 800
HEIGHT = 600
FOCAL_LENGTH = 500

SMPL_JOINT_NAMES = [
    "pelvis",
    "left_hip",
    "right_hip",
    "spine1",
    "left_knee",
    "right_knee",
    "spine2",
    "left_ankle",
    "right_ankle",
    "spine3",
    "left_foot",
    "right_foot",
    "neck",
    "left_collar",
    "right_collar",
    "head",
    "left_shoulder",
    "right_shoulder",
    "left_elbow",
    "right_elbow",
    "left_wrist",
    "right_wrist",
    "left_hand",
    "right_hand",
]

JOINT_MAP = {
    # 'BVH joint name': 'SMPL joint index'
    'Hips': SMPL_JOINT_NAMES.index('pelvis'),
    'Chest': SMPL_JOINT_NAMES.index('spine1'),
    'Chest2': SMPL_JOINT_NAMES.index('spine2'),
    'Chest3': SMPL_JOINT_NAMES.index('spine3'),
    # 'Chest4': SMPL_JOINT_NAMES.index('spine4'),
    'Neck': SMPL_JOINT_NAMES.index('neck'),
    'Head': SMPL_JOINT_NAMES.index('head'),
    'LeftCollar': SMPL_JOINT_NAMES.index('left_collar'),
    'LeftShoulder': SMPL_JOINT_NAMES.index('left_shoulder'),
    'LeftElbow': SMPL_JOINT_NAMES.index('left_elbow'),
    'LeftWrist': SMPL_JOINT_NAMES.index('left_wrist'),
    'RightCollar': SMPL_JOINT_NAMES.index('right_collar'),
    'RightShoulder': SMPL_JOINT_NAMES.index('right_shoulder'),
    'RightElbow': SMPL_JOINT_NAMES.index('right_elbow'),
    'RightWrist': SMPL_JOINT_NAMES.index('right_wrist'),
    'LeftHip': SMPL_JOINT_NAMES.index('left_hip'),
    'LeftKnee': SMPL_JOINT_NAMES.index('left_knee'),
    'LeftAnkle': SMPL_JOINT_NAMES.index('left_ankle'),
    'LeftToe': SMPL_JOINT_NAMES.index('left_foot'), #(Note: SMPL does not have separate toe joints)
    'RightHip': SMPL_JOINT_NAMES.index('right_hip'),
    'RightKnee': SMPL_JOINT_NAMES.index('right_knee'),
    'RightAnkle': SMPL_JOINT_NAMES.index('right_ankle'),
    'RightToe': SMPL_JOINT_NAMES.index('right_foot'), #(Note: SMPL does not have separate toe joints)
}

def bvh_to_smpl(bvh_file):
    with open(bvh_file, 'r') as f:
        mocap = Bvh(f.read())
    num_frames = 100 #mocap.nframes

    num_frames_downsampled = math.ceil(num_frames)

    smpl_poses = np.zeros((num_frames_downsampled, 72))
    smpl_trans = np.zeros((num_frames_downsampled, 3))
    bvh_joint_names = set(mocap.get_joints_names())
    # print(bvh_joint_names)

    rotation_correction = R.from_euler('XYZ', [0, 0, 0], degrees=True)

    for i in range(0, num_frames, 1):
        print('Processing frame {}/{}'.format(i, num_frames), end='\r')
        for joint_name, joint_index in JOINT_MAP.items():
            # print(joint_name, joint_index)

            rotation = R.from_euler('ZXY', mocap.frame_joint_channels(i, joint_name, ['Zrotation', 'Xrotation', 'Yrotation']), degrees=True)

            if joint_name == 'Hips':
                # rotation = rotation * rotation_correction
                rotation = rotation_correction * rotation

            smpl_poses[i, 3 * joint_index:3 * (joint_index + 1)] = rotation.as_rotvec()

            if joint_name == 'Hips':
                x, y, z = mocap.frame_joint_channels(i, joint_name, ['Xposition', 'Yposition', 'Zposition'])
                smpl_trans[i] = np.array([x, -z, y])

    # mirror
    # smpl_trans[:, 1] *= -1

    scale_factor = 0.009
    smpl_trans *= scale_factor

    return smpl_trans, smpl_poses

import pickle

def save_pkl(output_file, smpl_trans, smpl_poses, gender='female', model_type='smpl', frame_rate=30, smpl_scaling=1.0):
    data = {
        'smpl_trans': smpl_trans.astype(np.float32),
        'smpl_poses': smpl_poses.astype(np.float32),
        'smpl_scaling' : np.array([smpl_scaling], dtype=np.float32), 
    }
    with open(output_file, 'wb') as f:
        pickle.dump(data, f)

def generate_smpl(model, trans, poses):
    comp_device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

    model = model.to(comp_device)
    fId = 0
    root_orient = torch.Tensor(poses[:, :3]).to(comp_device)
    pose_body = torch.Tensor(poses[:, 3:]).to(comp_device)
    transl = torch.Tensor(trans[:]).to(comp_device)

    out = model(pose_body=pose_body, pose_hand=None, root_orient=root_orient)
    # joint_loc = body.Jtr[0] + transl

    return out, transl

if __name__ == '__main__':
    model_path = './smpl/SMPL_NEUTRAL.pkl'
    smpl_model = SMPL(model_path=model_path, num_betas=10, use_pca=False, use_face_contour=True, batch_size=100).cuda()

    fps = 30

    bvh_file = './BentForward_FW.bvh'
    output_file = './BentForward_FW.pkl'

    smplx_trans, smplx_poses = bvh_to_smpl(bvh_file)
    out, trans = generate_smpl(smpl_model, smplx_trans, smplx_poses)

    renderer = pyrender.OffscreenRenderer(viewport_width=WIDTH, viewport_height=HEIGHT)
    material = pyrender.MetallicRoughnessMaterial(
            metallicFactor=0.0,
            alphaMode='OPAQUE',
            baseColorFactor=(*(1.0, 1.0, 0.5), 1.0))
    scene = pyrender.Scene(bg_color=[*(0, 0, 0), 0.0],
                               ambient_light=(0.3, 0.3, 0.3))
    frames = []
    vertices = out.vertices.cpu().detach().numpy()
    for i in tqdm(range(out.vertices.shape[0])):
        tri_mesh = trimesh.Trimesh(vertices[i],
                                   smpl_model.faces)

        side_rot = trimesh.transformations.rotation_matrix(
                np.radians(90), [0, 1, 0])
        top_rot = trimesh.transformations.rotation_matrix(
                np.radians(-45), [0, 1, 0])
        tri_mesh.apply_transform(side_rot)
        tri_mesh.apply_transform(top_rot)

        mesh = pyrender.Mesh.from_trimesh(tri_mesh, material=material)

        camera_pose = np.eye(4)
        camera_pose[:3, 3] = trans.cpu()[i] + np.array([0, 0, 2]) 
        camera_pose[:3, :3] = euler_to_rotation_matrix((0, 0, -90))
        camera_center = [HEIGHT / 2, WIDTH / 2]
        camera = pyrender.IntrinsicsCamera(fx=FOCAL_LENGTH, fy=FOCAL_LENGTH,
                                           cx=camera_center[0], cy=camera_center[1], zfar=1e12)

        scene.clear()
        scene.add(camera, pose=camera_pose)
        scene.add(mesh)

        light_nodes = create_raymond_lights()
        for node in light_nodes:
            scene.add_node(node)

        color, depth = renderer.render(scene)
        frames.append(color)

    print("Saving the generated frames as mp4 file...")
    imageio.mimsave(f'test.mp4', frames, fps=float(fps))

    # with open(bvh_file, 'r') as f:
    #     mocap = Bvh(f.read())
    #     frame_rate = 1.0 / mocap.frame_time
    # print('frame_rate: ', frame_rate)
    # save_pkl(output_file,smplx_trans,smplx_poses,frame_rate)

visualization code:

def euler_to_rotation_matrix(angles_in_degrees):
    """
    Convert Euler angles in degrees to a 3x3 rotation matrix.

    Parameters:
        angles_in_degrees (tuple): A tuple of three Euler angles (x, y, z) in degrees.

    Returns:
        np.ndarray: A 3x3 rotation matrix.
    """
    angles = np.radians(angles_in_degrees)
    x, y, z = angles

    Rx = np.array([[1, 0, 0],
                   [0, np.cos(x), -np.sin(x)],
                   [0, np.sin(x), np.cos(x)]])

    Ry = np.array([[np.cos(y), 0, np.sin(y)],
                   [0, 1, 0],
                   [-np.sin(y), 0, np.cos(y)]])

    Rz = np.array([[np.cos(z), -np.sin(z), 0],
                   [np.sin(z), np.cos(z), 0],
                   [0, 0, 1]])

    R = Rz @ Ry @ Rx
    return R

def create_raymond_lights():
    """
    Return raymond light nodes for the scene.
    """
    thetas = np.pi * np.array([1.0 / 6.0, 1.0 / 6.0, 1.0 / 6.0])
    phis = np.pi * np.array([0.0, 2.0 / 3.0, 4.0 / 3.0])

    nodes = []

    for phi, theta in zip(phis, thetas):
        xp = np.sin(theta) * np.cos(phi)
        yp = np.sin(theta) * np.sin(phi)
        zp = np.cos(theta)

        z = np.array([xp, yp, zp])
        z = z / np.linalg.norm(z)
        x = np.array([-z[1], z[0], 0.0])
        if np.linalg.norm(x) == 0:
            x = np.array([1.0, 0.0, 0.0])
        x = x / np.linalg.norm(x)
        y = np.cross(z, x)

        matrix = np.eye(4)
        matrix[:3,:3] = np.c_[x,y,z]
        nodes.append(pyrender.Node(
            light=pyrender.DirectionalLight(color=np.ones(3), intensity=1.0),
            matrix=matrix
        ))

    return nodes