How to get the position of Character in 3D space?

[SlimeVRX]

Thanks to the author team for this great work!

I fixed some bugs in Colab to make it work. I have detailed notes.

Thank you very much!

[SlimeVRX]

It takes about 1 hour to install all the libraries

With Colab Tesla T4, It takes 15-20 minutes for 5s 30 FPS video

[SlimeVRX]

https://user-images.githubusercontent.com/99313947/220333439-147ebeb8-a579-4ff7-8ec9-8d529492b088.mp4

https://user-images.githubusercontent.com/99313947/220333483-02d6fa0d-7b86-42ba-84c4-06eec284431c.mp4

[yhw-yhw]

Awesome! Can you submit a pull request of it?

We will release the hugging face demo soon.

[yhw-yhw]

BTW, you can change the body shape gender as 'male'. Now the gender that you are running with is 'female'.

[SlimeVRX]

Hi!

Your Colab link is not in Github, it's in Drive, I don't have permission to edit it. Please refer to my Colab

# Error 1: /content/SHOW/SHOW/video_filter/MMposer.py
# Line 12: 
#     det_config = os.path.join(mmpose_root,'demo\mmdetection_cfg/faster_rcnn_r50_fpn_coco.py')
# Replace: 
#     det_config = os.path.join(mmpose_root,'demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py')

# Error 2: Unable to download antelopev2.zip

# Error 3: RuntimeError: Subtraction, the `-` operator, with a bool tensor is not supported. If you are trying to invert a mask, use the `~` or `logical_not()` operator instead.
#          /usr/local/lib/python3.8/dist-packages/torchgeometry/core/conversions.py
# Line 302-304:         
#   mask_c1 = mask_d2 * (1 - mask_d0_d1)
#   mask_c2 = (1 - mask_d2) * mask_d0_nd1
#   mask_c3 = (1 - mask_d2) * (1 - mask_d0_nd1)
# Replace:
#   mask_c1 = mask_d2 * (~ mask_d0_d1)
#   mask_c2 = (~ mask_d2) * mask_d0_nd1
#   mask_c3 = (~ mask_d2) * (~ mask_d0_nd1)

[yhw-yhw]

OK, sorry about it, will update it in the github. Huge thanks.

[SlimeVRX]

BTW, you can change the body shape gender as 'male'. Now the gender that you are running with is 'female'.

Hi!

In models/smplx there are 2 models:

SMPLX_MALE_shape2019_exp2020.npz
SMPLX_NEUTRAL_2020_org.npz

Why is there a "female" gender in my demo?

[SlimeVRX]

I have another problem: The body is moving right but the fingers moving very strangely

https://user-images.githubusercontent.com/99313947/221088047-393be93f-6204-4b80-a0cd-8fcc882b332a.mp4

[SlimeVRX]

From file all.pkl

I can get the parameters:

vertices
joints
full_pose
global_orient
transl
v_shaped
betas
body_pose
left_hand_pose
right_hand_pose
expression
jaw_pose

jaw_pose (210, 3)
leye_pose (210, 3)
reye_pose (210, 3)
global_orient (210, 3)
body_pose (210, 63)
left_hand_pose (210, 12)
right_hand_pose (210, 12)

I got Poses from:

global_orient (3 -> 1,3)
body_pose (63 -> 21,3)
jaw (3 -> 1,3)
leye (3 -> 1,3)
reye (3 -> 1,3)
left_hand_pose (45 -> 15,3)
left_hand_pose (45 -> 15,3)

total (165 -> 55,3)

[SlimeVRX]

I have arranged it in the correct order below, refer to: Link

    HIPS = "Hips"                                   # "pelvis",
    LEFT_UPPER_LEG = "LeftUpperLeg"                 # "left_hip",
    RIGHT_UPPER_LEG = "RightUpperLeg"               # "right_hip",
    SPINE = "Spine"                                 # "spine1",
    LEFT_LOWER_LEG = "LeftLowerLeg"                 # "left_knee",
    RIGHT_LOWER_LEG = "RightLowerLeg"               # "right_knee",
    CHEST = "Chest"                                 # "spine2",
    LEFT_FOOT = "LeftFoot"                          # "left_ankle",
    RIGHT_FOOT = "RightFoot"                        # "right_ankle",
    UPPER_CHEST = "UpperChest"                      # "spine3",
    LEFT_TOES = "LeftToes"                          # "left_foot",
    RIGHT_TOES = "RightToes"                        # "right_foot",
    NECK = "Neck"                                   # "neck",
    LEFT_SHOULDER = "LeftShoulder"                  # "left_collar",
    RIGHT_SHOULDER = "RightShoulder"                # "right_collar",
    HEAD = "Head"                                   # "head",
    LEFT_UPPER_ARM = "LeftUpperArm"                 # "left_shoulder",
    RIGHT_UPPER_ARM = "RightUpperArm"               # "right_shoulder",
    LEFT_LOWER_ARM = "LeftLowerArm"                 # "left_elbow",
    RIGHT_LOWER_ARM = "RightLowerArm"               # "right_elbow",
    LEFT_HAND = "LeftHand"                          # "left_wrist",
    RIGHT_HAND = "RightHand"                        # "right_wrist",
    JAW = "Jaw"                                     # "jaw",
    LEFT_EYE = "LeftEye"                            # "left_eye_smplhf",
    RIGHT_EYE = "RightEye"                          # "right_eye_smplhf",
    LEFT_INDEX_PROXIMAL = "LeftIndexProximal"           # "left_index1",
    LEFT_INDEX_INTERMEDIATE = "LeftIndexIntermediate"   # "left_index2",
    LEFT_INDEX_DISTAL = "LeftIndexDistal"               # "left_index3",
    LEFT_MIDDLE_PROXIMAL = "LeftMiddleProximal"         # "left_middle1",
    LEFT_MIDDLE_INTERMEDIATE = "LeftMiddleIntermediate" # "left_middle2",
    LEFT_MIDDLE_DISTAL = "LeftMiddleDistal"             # "left_middle3",
    LEFT_LITTLE_PROXIMAL = "LeftLittleProximal"         # "left_pinky1",
    LEFT_LITTLE_INTERMEDIATE = "LeftLittleIntermediate" # "left_pinky2",
    LEFT_LITTLE_DISTAL = "LeftLittleDistal"             # "left_pinky3",
    LEFT_RING_PROXIMAL = "LeftRingProximal"             # "left_ring1",
    LEFT_RING_INTERMEDIATE = "LeftRingIntermediate"     # "left_ring2",
    LEFT_RING_DISTAL = "LeftRingDistal"                 # "left_ring3",
    LEFT_THUMB_PROXIMAL = "LeftThumbProximal"           # "left_thumb1",
    LEFT_THUMB_INTERMEDIATE = "LeftThumbIntermediate"   # "left_thumb2",
    LEFT_THUMB_DISTAL = "LeftThumbDistal"               # "left_thumb3",
    RIGHT_INDEX_PROXIMAL = "RightIndexProximal"             # "right_index1",
    RIGHT_INDEX_INTERMEDIATE = "RightIndexIntermediate"     # "right_index2",
    RIGHT_INDEX_DISTAL = "RightIndexDistal"                 # "right_index3",
    RIGHT_MIDDLE_PROXIMAL = "RightMiddleProximal"           # "right_middle1",
    RIGHT_MIDDLE_INTERMEDIATE = "RightMiddleIntermediate"   # "right_middle2",
    RIGHT_MIDDLE_DISTAL = "RightMiddleDistal"               # "right_middle3",
    RIGHT_LITTLE_PROXIMAL = "RightLittleProximal"           # "right_pinky1",
    RIGHT_LITTLE_INTERMEDIATE = "RightLittleIntermediate"   # "right_pinky2",
    RIGHT_LITTLE_DISTAL = "RightLittleDistal"               # "right_pinky3",
    RIGHT_RING_PROXIMAL = "RightRingProximal"               # "right_ring1",
    RIGHT_RING_INTERMEDIATE = "RightRingIntermediate"       # "right_ring2",
    RIGHT_RING_DISTAL = "RightRingDistal"                   # "right_ring3",
    RIGHT_THUMB_PROXIMAL = "RightThumbProximal"             # "right_thumb1",
    RIGHT_THUMB_INTERMEDIATE = "RightThumbIntermediate"     # "right_thumb2",
    RIGHT_THUMB_DISTAL = "RightThumbDistal"                 # "right_thumb3",

[lithiumice]

BTW, you can change the body shape gender as 'male'. Now the gender that you are running with is 'female'.

Hi!

In models/smplx there are 2 models:

SMPLX_MALE_shape2019_exp2020.npz
SMPLX_NEUTRAL_2020_org.npz

Why is there a "female" gender in my demo?

The results of this NEUTRAL model sometimes become closer to FEMALE, you can change it to MALE in (https://github.com/yhw-yhw/SHOW/blob/22c36e5030e64d3a9260b5d0b71c89406172deae/configs/base/model_smplx_config.py#L10)

There are some miss code in previous codebase: (https://github.com/yhw-yhw/SHOW/blob/22c36e5030e64d3a9260b5d0b71c89406172deae/stage2_main.py#L828). And the final pkl file of results should be ./test/demo_video/ours_exp/final_all.pkl rather than ./test/demo_video/ours/all.pkl

[SlimeVRX]

Hi!

Thank you very much, I will try again.

[SlimeVRX]

I found the cause of the problem

from all.pkl file

jaw_pose (210, 3)
leye_pose (210, 3)
reye_pose (210, 3)
global_orient (210, 3)
body_pose (210, 63)
left_hand_pose (210, 12)
right_hand_pose (210, 12)

vertices
joints
full_pose
global_orient
transl
v_shaped
betas
body_pose
left_hand_pose
right_hand_pose
expression
jaw_pose

Figure 1 use "vertices" parameters should be correct

Figure 2 use "bone" parameters

Reason:

left_hand_pose (210, 12)
right_hand_pose (210, 12)

12 -> (4,3) which means the whole hand has only 4 points estimated

there should be 45 -> (15,3), the whole hand has 15 points estimated -> 5 fingers

[SlimeVRX]

I will find information from Pymaf-X model

About the hand, besides the vertices information, is there enough bone information of 15 points? 5 fingers?

Thank you!

[lithiumice]

I will find information from Pymaf-X model

About the hand, besides the vertices information, is there enough bone information of 15 points? 5 fingers?

Thank you!

In fact, the dimension of the parameter of hand here is (bs, 12), where the 12 means the PCA dimension, not the axis angle of hand, the method of converting hand PCA to axis angle can be referred to here

[SlimeVRX]

Thank you for the information!

I will follow your instructions

Thank you very much!

[lithiumice]

for example:

import smplx

smplx_cfg=dict(
    model_path='path_to_smplx_model'
    model_type= 'smplx',
    gender= 'neutral',
    use_face_contour= True,
    use_pca= True,
    flat_hand_mean= False,
    use_hands= True,
    use_face= True,
    num_pca_comps= 12,
    num_betas= 300,
    num_expression_coeffs= 100,
)
body_model = smplx.create(**smplx_cfg)
model_output = body_model(
    return_verts=True,
    return_full_pose=True,

    betas=body_params['betas'],
    jaw_pose=body_params['jaw_pose'],
    leye_pose=body_params['leye_pose'],
    reye_pose=body_params['reye_pose'],
    expression=body_params['expression'],

    transl=body_params['transl'],
    body_pose=body_params['body_pose_axis'],
    global_orient=body_params['global_orient'],
    left_hand_pose=body_params['left_hand_pose'], # note: left_hand_pose shape is (batch_size, 12)
    right_hand_pose=body_params['right_hand_pose'],
) 
axis_lhand = model_output.left_hand_pose# note: axis_lhand shape is (batch_size, 45)

[SlimeVRX]

Thank you! I will try it

[SlimeVRX]

Hi!

It worked. I fixed the fingers

model_output = body_model(return_verts=True, return_full_pose=True, **smplx_params)
full_pose = model_output.full_pose.detach().cpu().numpy().squeeze()
full_pose.shape (157, 165)

165 -> (55,3) 55 points estimated

https://user-images.githubusercontent.com/99313947/221391677-aebbe7a5-214b-4634-a531-5e44b266299c.mp4

https://user-images.githubusercontent.com/99313947/221391682-286d0ea1-f4d7-4cc4-847a-d8725ee67946.mp4

Thank you very much!

[SlimeVRX]

Hi!

I got 3D Poses, I need 3D Trans is the position of Character in 3D space How to calculate this?

Thank you very much!

[SlimeVRX]

Hi!

I use the "transl" parameter which is the Character's position in 3D

Then I send the data to Unreal

https://user-images.githubusercontent.com/99313947/221740004-43c6cc94-b66a-477e-99eb-f6237da6b5d0.mp4

[yhw-yhw]

Can you add facial motion in this demo?

[yhw-yhw]

BTW, @SlimeVRX do you want to create an Unreal/Unity Add-on for TalkSHOW? We believe this model is very useful to generate holistic body motion from an input audio only.

[SlimeVRX]

Hi,

I need face data format like ARkit, I don't know how to convert Face data from SMPL to ARkit. There is a discussion about converting Face data from SMPL to ARkit here

BTW, @SlimeVRX do you want to create an Unreal/Unity Add-on for TalkSHOW? We believe this model is very useful to generate holistic body motion from an input audio only.

If you need the plugin for Unreal, I can send it to you

[SlimeVRX]

https://user-images.githubusercontent.com/99313947/232279762-791388d5-1521-4b3b-9b5b-d0222cc0c090.mp4

[SlimeVRX]

https://user-images.githubusercontent.com/99313947/232280898-190458f4-6322-40f1-95fd-366cce147405.mp4

[SlimeVRX]

https://user-images.githubusercontent.com/99313947/232280992-d15bf565-3746-4d67-a5c2-3a49cb874255.mp4

[SlimeVRX]

Hi @yhw-yhw

Wonderdynamics is a new trend in video creation:

• Remove people from the background,
• Select characters
• Tracking camera
• Render light

I want to use SHOW as core to Mocap. I tested it and the result is very good!

https://user-images.githubusercontent.com/99313947/232281048-787d96e9-25fa-41ef-b286-3d0af9505599.mp4