Can this code be used for smplx?

[Darius-H]

Can this code be used for smplx deformation? Since it's cuda code here which I'm not familiar with, I'm not sure if it can be used for other init_bones. And I wonder how init_bones are selected? Why not use all the 24 bones, any advice? https://github.com/tijiang13/InstantAvatar/blob/4348c1e5a38627a8096d288072896e281577d4f5/instant_avatar/deformers/fast_snarf/deformer_torch.py#L28

[tijiang13]

Hi Darius-H,

While it's possible, the current naive implementation will consume a huge amount of GPU memory. The core idea of SNARF is to deform a point with all available bones, resulting in a 24x increase in GPU load (for SMPL). And this will become a bigger problem with SMPLX as it has much more bones. If you want to adapt it for SMPLX you will need to add a filter to reduce the candidate bones before passing it to SNARF (which should be relatively easy).

For init_bones we pick the more "important" bones (like legs/arms). You could use 24 bones which will improve the quality of hands and feet, but will also slow down the training a bit.

Best, Tianjian

[sora158]

I have tried change smpl to smplx, and some issues really confuse me. Like, the model don't have hands. You know, smplx models have more init bones than smpl, like 55 bones vs 23 bones(most of them are hand joints), so I change the init bones to 55, all of joints of smplx are considered. And below is the result: hands are gone!

[sora158]

And next I tried to visuilize the lbs_weights >0.9, and I found smplx_lbs_weights which compared to smpl, most of hands lbs weights are less than 0.9, so most of hand areas are zero. first is smpl lbs weights:

second is smplx lbs_weights:

What I have done is just switch the body pose of smpl to smplx: smplx["body_pose"] = smpl["body_pose"][:,:63] #smplx add MANO to represent hand, so smplx body pose is [1,63], two hand joints are taken off. and I make hand poses/expressions are zeros. the results are shown above, no hands. What's more, I also tried use my own way to get smplx parameters, however, the results are even worse, I wonder the you have some requirements in world coordinates, right? Because I visuilize the smplx model and rays, they are all good. That's my second question. I really confused by the no hands issue, can you give me some hint? Best wishes!

[Darius-H]

And next I tried to visuilize the lbs_weights >0.9, and I found smplx_lbs_weights which compared to smpl, most of hands lbs weights are less than 0.9, so most of hand areas are zero. first is smpl lbs weights:

second is smplx lbs_weights:

What I have done is just switch the body pose of smpl to smplx: smplx["body_pose"] = smpl["body_pose"][:,:63] #smplx add MANO to represent hand, so smplx body pose is [1,63], two hand joints are taken off. and I make hand poses/expressions are zeros. the results are shown above, no hands. What's more, I also tried use my own way to get smplx parameters, however, the results are even worse, I wonder the you have some requirements in world coordinates, right? Because I visuilize the smplx model and rays, they are all good. That's my second question. I really confused by the no hands issue, can you give me some hint? Best wishes!

Did you try to visualize your smplx model with the original image? Your way to create smplx params may result in terrible error. As smpl and smplx model are trained sperately, simply copy smpl body pose to smplx and remain smpl's betas and transl can result in smplx body unalignment with the original image. And InstantAvatar's performance seriously depends on the accuracy of smplx params, make hand poses/expressions zeros will result in the hand points not correctly deforming to canonical space. Maybe you should try this: https://github.com/HongwenZhang/PyMAF-X.

[sora158]

I have tried [PyMAF-X], also don't have hands, and I also visulized the smplx model: rays/models/images are well aligned, everything are fine except hands, which really confuses me. So I tried to simply use provided smpl parameters. Maybe I shouldn't set the hand poses to be zero, I will try again later, Thank you!

[sora158]

I used anim_nerftrain.npz, add defualt hand poses, all the other things are the same as the PeopleSnapShotDataset(), and the first epoch shows result like below:  It looks really like smplx version of self.lbs_voxel_final . So maybe you can try switch smpl to smplx and see if the results are the same.

[sora158]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

[tijiang13]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Sorry for the confusion. I forgot to mention that some cuda code includes hardcoded hyperparameters like in precompute.cu. Also some comments in the cuda code (like the shape of tensors) are outdated and need updating.

Best, Tianjian

[sora158]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Sorry for the confusion. I forgot to mention that some cuda code includes hardcoded hyperparameters like in precompute.cu. Also some comments in the cuda code (like the shape of tensors) are outdated and need updating.

Best, Tianjian

Thanks for reply, I'm trying to use PyMAF-X to predict smplx parameters. When I use camera intrinstics and extrinstics from PyMAF-X， and use your code to calculate rays:

camera_data = np.load('PyMAF-X/camera.npz')
fx = camera_data['fx']
fy = camera_data['fy']
cx = camera_data['cx']
cy = camera_data['cy']
camera_pose = camera_data['camera_pose'] #4*4

def make_rays(K, c2w, H, W):
    xy = get_ray_directions(H, W).reshape(-1, 3).astype(np.float32)
    d_c = xy @ np.linalg.inv(K).T
    d_w = d_c @ c2w[:3, :3].T
    d_w = d_w / np.linalg.norm(d_w, axis=1, keepdims=True)
    o_w = np.tile(c2w[:3, 3], (len(d_w), 1))
    o_w = o_w.reshape(H, W, 3)
    d_w = d_w.reshape(H, W, 3)
    return o_w.astype(np.float32), d_w.astype(np.float32)

Though, rays are not right. However, they are right when visualized in pyrender. I wonder if you give me some hints about how to get right rays_o and rays_d when have camera intrinstics and extrinstics? Best wishes!

[tijiang13]

Hi Sora158,

We use OpenCV coordinates, while it's possible that they are using OpenGL coordinates. To check the accuracy of the estimated poses, you can utilize the visualization tool to confirm if the overlay aligns correctly (check this link). The code above uses SMPL model but you can switch to SMPLX easily by changing this line to model_type='smplx'

Best, Tianjian

[sora158]

Thanks for your kindness reply! It really helps a lot! : )

[Darius-H]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Could you tell me what part of the cuda code I need to change when coverting to smplx? I don't know cuda much. Or maybe provide an updated code. Thanks very much. @sora158 @tijiang13

[sora158]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Could you tell me what part of the cuda code I need to change when coverting to smplx? I don't know cuda much. Or maybe provide an updated code. Thanks very much. @sora158 @tijiang13

change the code in precompute.cu #Line54

      for (index_t j = 0; j < 55; j++) {

[tijiang13]

Hi Darius-H,

In my opinion this line assumes 24 joints and after changing this should already work.

Best, Tianjian

[sora158]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Could you tell me what part of the cuda code I need to change when coverting to smplx? I don't know cuda much. Or maybe provide an updated code. Thanks very much. @sora158 @tijiang13

can you give me some hints about how to compute the rays_o and rays_d in pymafx dataset? I tried to visualize,overlay aligns correctly, but the results are bad, thank you

[tijiang13]

can you give me some hints about how to compute the rays_o and rays_d in pymafx dataset? I tried to visualize,overlay aligns correctly, but the results are bad, thank you

I have never tried that before but I don't think there should be anything special about computing rays. The generation of rays should solely depend on the intrinsic and extrinsic parameters of cameras so it should be irrelevant to SMPL/SMPLX.

[tijiang13]

But I think that it's non-trivial to adapt to SMPLX. For example, we represent the lbs field with a low-resolution volume and it might cause problems for fingers and expressions.

[Darius-H]

But I think that it's non-trivial to adapt to SMPLX. For example, we represent the lbs field with a low-resolution volume and it might cause problems for fingers and expressions.

I tried to use smplx. Fast-snarf did not show better results than vanilla smplx deformers, and fingers are still difficult to distinguish.

[sora158]

But I think that it's non-trivial to adapt to SMPLX. For example, we represent the lbs field with a low-resolution volume and it might cause problems for fingers and expressions.

I tried to use smplx. Fast-snarf did not show better results than vanilla smplx deformers, and fingers are still difficult to distinguish.

can you tell me how you calculate rays in pymafx dataset? I tried to convert pyrender coordinates to opencv coordinates and compute rays, didn't work out

[tijiang13]

I tried to use smplx. Fast-snarf did not show better results than vanilla smplx deformers, and fingers are still difficult to distinguish.

I think that's somewhat expected. The texture of fingers look similar and if the initialization is not good enough then anyway it will be difficult to produce reasonable deformation. Most recent works like AvatarRex/ECON etc. either use mano mesh directly or have some special treatment for hand (like utilizing a hand detector to extract hand bbox and add more samples within that region)

[Darius-H]

But I think that it's non-trivial to adapt to SMPLX. For example, we represent the lbs field with a low-resolution volume and it might cause problems for fingers and expressions.

I tried to use smplx. Fast-snarf did not show better results than vanilla smplx deformers, and fingers are still difficult to distinguish.

can you tell me how you calculate rays in pymafx dataset? I tried to convert pyrender coordinates to opencv coordinates and compute rays, didn't work out

I didn't change the code of calculating rays. But it takes me much time to calculate the right smplx translation and camera translation from pymafx. Give me your email i can send it to you.

[sora158]

But I think that it's non-trivial to adapt to SMPLX. For example, we represent the lbs field with a low-resolution volume and it might cause problems for fingers and expressions.

I tried to use smplx. Fast-snarf did not show better results than vanilla smplx deformers, and fingers are still difficult to distinguish.

can you tell me how you calculate rays in pymafx dataset? I tried to convert pyrender coordinates to opencv coordinates and compute rays, didn't work out

I didn't change the code of calculating rays. But it takes me much time to calculate the right smplx translation and camera translation from pymafx. Give me your email i can send it to you.

my email is 1085267562@qq.com I take days to think the right translation but still didn't work out, Thank you for your kindness!

[muximuxi]

problem resolved. You should change the code in precompute.cu and change all the parameters from smpl to smplx ,including joints .etc

Can you tell me which code file to modify the joints? , I change the refine-smpl.py when I process the data, and get this error: File "scripts/custom/refine-smpl.py", line 279, in main(args.data_dir, args.gender, args.keypoints_threshold, args.silhouette, args.downscale) File "/miniconda3/envs/instant/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, kwargs) File "scripts/custom/refine-smpl.py", line 209, in main optimize(optimizer, closure, max_iter=200) File "scripts/custom/refine-smpl.py", line 26, in optimize loss = optimizer.step(closure) File "/miniconda3/envs/instant/lib/python3.8/site-packages/torch/optim/optimizer.py", line 280, in wrapper out = func(*args, *kwargs) File "/miniconda3/envs/instant/lib/python3.8/site-packages/torch/optim/optimizer.py", line 33, in _use_grad ret = func(self, args, kwargs) File "/miniconda3/envs/instant/lib/python3.8/site-packages/torch/optim/adam.py", line 121, in step loss = closure() File "scripts/custom/refine-smpl.py", line 192, in closure smpl_output = body_model(*params) File "/miniconda3/envs/instant/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "/miniconda3/envs/instant/lib/python3.8/site-packages/smplx/body_models.py", line 1209, in forward body_pose.reshape(-1, self.NUM_BODY_JOINTS, 3), RuntimeError: shape '[-1, 21, 3]' is invalid for input of size 7038