blender load pose?

[kaisunc]

the pkl output from fit_single_image have a dimension of (1,32) and looking at the code in the blender plugin, it's looking for a shape of (1,63)

            if body_pose.shape != (1, NUM_SMPLX_BODYJOINTS * 3):
                print(f"Invalid body pose dimensions: {body_pose.shape}")

Looking through the docs, it seems that human body prior Vposer is outputing as poZ encoding? And the blender plugin is looking for smpl body pose encoding?

[thibaut-evrard]

I ran into the same issue whilst trying to import the pkl output to blender. From what I can understand, the issue comes from the fact that the body_pose data in the .pkl output is somehow encoded into a different format than the one Blender is expecting e.g. [ x0 ,y0, z0, x1, y1, z1 ... ]

I was able to make the body pose work in blender by changing a line in fit_single_frame.py from: result['body_pose'] = pose_embedding.detach().cpu().numpy() to: result['body_pose'] = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63))

As I mentioned, this helps with putting body_pose into the right format, however, there is another formating problem in the left_hand_pose and right_hand_pose arguments of the .pkl. still havent found how to decode this bit. If anyone has an idea, that would be much appreciated :)

[zhan0502]

Hi,

Thank you for sharing. I have been trying to load the pose output from smplify-x to blender and encounter the same issue. Has anyone been able to make the mapping works?

Thanks. :)

[UtsaChattopadhyay]

Hi... So I'm also getting the same issue... others have commented that the problem is with the add-on file... but nothing more.. any help will be appreciated... Thank you. Invalid body pose dimensions: (1, 32) was fixed by changing from: result['body_pose'] = pose_embedding.detach().cpu().numpy() to: result['body_pose'] = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63)) however, there is another formating problem in the left_hand_pose and right_hand_pose arguments of the .pkl. If anyone has an idea, that would be much appreciated :)))

[haiderasad]

hi @UtsaChattopadhyay @zhan0502 @thibaut-evrard any update on the hands formatting? the body is working fine in blender thanks to @thibaut-evrard but the hand and fingers are still not replicating

[thibaut-evrard]

Hi, @haiderasad, sorry for the late reply. I managed to achieve this in the end, but unfortunately the work is on another machine I do not have access to at the moment.

As I remember, I was able to fix this by looking at the code of the SMPLify Blender add-on and the code of this repo that manages the export. the hands problem is very similar to what was happening to the body as I remember. Just the blender add-on receiving data formatted in an incorrect way. I hope this helps for now. I will try and get access to the machine I was able to run the code on to share the solution.

[haiderasad]

Hi, @haiderasad, sorry for the late reply. I managed to achieve this in the end, but unfortunately the work is on another machine I do not have access to at the moment.

As I remember, I was able to fix this by looking at the code of the SMPLify Blender add-on and the code of this repo that manages the export. the hands problem is very similar to what was happening to the body as I remember. Just the blender add-on receiving data formatted in an incorrect way. I hope this helps for now. I will try and get access to the machine I was able to run the code on to share the solution.

@thibaut-evrard thank for this insight if you do get access to the machine kindly let me know the hand fix code snippet, would much appreciate it thanks

[thevoidsteve]

你好，@haiderasad， 这么晚才回复很抱歉。 我最终设法实现了这一点，但不幸的是，这项工作是在我目前无法访问的另一台机器上进行的。

我记得，我能够通过查看 SMPLify Blender 附加组件的代码和管理导出的此 repo 的代码来解决此问题。手的问题与我记忆中发生在身体上的问题非常相似。只是搅拌机插件接收格式不正确的数据。 我希望这对现在有帮助。我将尝试访问我能够运行代码的机器以共享解决方案。

Hi, @haiderasad, sorry for the late reply. I managed to achieve this in the end, but unfortunately the work is on another machine I do not have access to at the moment. As I remember, I was able to fix this by looking at the code of the SMPLify Blender add-on and the code of this repo that manages the export. the hands problem is very similar to what was happening to the body as I remember. Just the blender add-on receiving data formatted in an incorrect way. I hope this helps for now. I will try and get access to the machine I was able to run the code on to share the solution.

@thibaut-evrard thank for this insight if you do get access to the machine kindly let me know the hand fix code snippet, would much appreciate it thanks

Have you solved the problem of hand posture loading? Can you tell me the solution? Thank you very much!

[a7mad7aydar]

That solved my problem! Even though there is still a problem with the hands, it got imported! Many thanks!

I ran into the same issue whilst trying to import the pkl output to blender. From what I can understand, the issue comes from the fact that the body_pose data in the .pkl output is somehow encoded into a different format than the one Blender is expecting e.g. [ x0 ,y0, z0, x1, y1, z1 ... ]

I was able to make the body pose work in blender by changing a line in fit_single_frame.py from: result['body_pose'] = pose_embedding.detach().cpu().numpy() to: result['body_pose'] = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63))

As I mentioned, this helps with putting body_pose into the right format, however, there is another formating problem in the left_hand_pose and right_hand_pose arguments of the .pkl. still havent found how to decode this bit. If anyone has an idea, that would be much appreciated :)

[long10024070]

I found a solution for left_hand_pose and right_hand_pose argument problem. The problem is that result object DOES NOT CONTAIN left_hand_pose and right_hand_pose (also body_pose) argument in the format of SMPLX output. I read the code in SMPLX demo and found that we can receive SMPLX output in the correct shape (body: (1,63), hand: (1,45)) through the forward() method or ()operator of the model object (body_model in fit_single_frame function). There is an new problem with the forward() method is that it requires body_model paramenter as this paramenter doesn't exist in body_model when the use_vposer flag is set as True. Thank to the previous comment of @thibaut-evrard, I can get the body_pose value with vposer and use it as an input paramenter for the forward() method in order to get left_hand_pose and right_hand_pose. In short, this is my code to get both body_pose, left_hand_pose and right_hand_pose:

if use_vposer:
    body_pose = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63))
    output_pose = body_model.forward(body_pose=body_pose)
    left_hand_pose = output_pose.left_hand_pose.detach().cpu().numpy()
    right_hand_pose = output_pose.right_hand_pose.detach().cpu().numpy()
else:
    output_pose = body_model.forward()
    body_pose = output_pose.body_pose.detach().cpu().numpy()
    left_hand_pose = output_pose.left_hand_pose.detach().cpu().numpy()
    right_hand_pose = output_pose.right_hand_pose.detach().cpu().numpy()

However, I'm not sure that this is enough for getting input data for load_pose in Blender, because I don't know why body_model.named_parameters() in result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters()}) returns a far different output, compared to body_model.forward(). Therefore, I have made a little change in fit_single_frame.py. From:

result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters()})
if use_vposer:
    result['body_pose'] = pose_embedding.detach().cpu().numpy()

to:

body_model_forward = body_model.forward(body_pose=vposer.decode(pose_embedding).get( 'pose_body') if use_vposer else None)
result.update({key: val.detach().cpu().numpy() for key, val in body_model_forward.items() if val is not None})
result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters() if key not in result})
# reshape body_pose for SMPLX-blender
result['body_pose'] = result['body_pose'].reshape((1,-1))

Note: The key to solve this problem is the forward() method which returns an output object (SMPLOuput, SMPLHOutput, SMPLXOuput, eg, which is dependent on your model_type input flag) containing all result paramenters you need. However, forward() doesn't work without paramenter when the use_vposer flag is turned on. I think that is why @vchoutas used named_parameters() instead, but it caused the shape of paramenters in result to be different from their original format.

Bonus: The format of hand/body_pose is truly [ x0 ,y0, z0, x1, y1, z1 ... ] as @thibaut-evrard wrote. In addition, each set of three numbers controls a corresponding bone in Blender. They define a vector, which directs the bone and then rotate by an angle equal to the vector's length (in radian, I guess). Athough I know the controlling rule and postion of joints in 3D via issus #106, I don't know how to calculate orientation for Blender load_pose, specially the rotation. Luckily, the code above saved me. This is my result:	Input	Output 1	Output 2

I captured both output poses in the default front view. In the first output, the character is complety upside down. I manually set the global_orient paramenter to [[0,0,0]] and received the second output.

[meimeibuchikafei]

I found a solution for left_hand_pose and right_hand_pose argument problem. The problem is that result object DOES NOT CONTAIN left_hand_pose and right_hand_pose (also body_pose) argument in the format of SMPLX output. I read the code in SMPLX demo and found that we can receive SMPLX output in the correct shape (body: (1,63), hand: (1,45)) through the forward() method or ()operator of the model object (body_model in fit_single_frame function). There is an new problem with the forward() method is that it requires body_model paramenter as this paramenter doesn't exist in body_model when the use_vposer flag is set as True. Thank to the previous comment of @thibaut-evrard, I can get the body_pose value with vposer and use it as an input paramenter for the forward() method in order to get left_hand_pose and right_hand_pose. In short, this is my code to get both body_pose, left_hand_pose and right_hand_pose:

if use_vposer:
    body_pose = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63))
    output_pose = body_model.forward(body_pose=body_pose)
    left_hand_pose = output_pose.left_hand_pose.detach().cpu().numpy()
    right_hand_pose = output_pose.right_hand_pose.detach().cpu().numpy()
else:
    output_pose = body_model.forward()
    body_pose = output_pose.body_pose.detach().cpu().numpy()
    left_hand_pose = output_pose.left_hand_pose.detach().cpu().numpy()
    right_hand_pose = output_pose.right_hand_pose.detach().cpu().numpy()

However, I'm not sure that this is enough for getting input data for load_pose in Blender, because I don't know why body_model.named_parameters() in result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters()}) returns a far different output, compared to body_model.forward(). Therefore, I have made a little change in fit_single_frame.py. From:

result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters()})
if use_vposer:
    result['body_pose'] = pose_embedding.detach().cpu().numpy()

to:

body_model_forward = body_model.forward(body_pose=vposer.decode(pose_embedding).get( 'pose_body') if use_vposer else None)
result.update({key: val.detach().cpu().numpy() for key, val in body_model_forward.items() if val is not None})
result.update({key: val.detach().cpu().numpy() for key, val in body_model.named_parameters() if key not in result})
# reshape body_pose for SMPLX-blender
result['body_pose'] = result['body_pose'].reshape((1,-1))

Note: The key to solve this problem is the forward() method which returns an output object (SMPLOuput, SMPLHOutput, SMPLXOuput, eg, which is dependent on your model_type input flag) containing all result paramenters you need. However, forward() doesn't work without paramenter when the use_vposer flag is turned on. I think that is why @vchoutas used named_parameters() instead, but it caused the shape of paramenters in result to be different from their original format.

Bonus: The format of hand/body_pose is truly [ x0 ,y0, z0, x1, y1, z1 ... ] as @thibaut-evrard wrote. In addition, each set of three numbers controls a corresponding bone in Blender. They define a vector, which directs the bone and then rotate by an angle equal to the vector's length (in radian, I guess). Athough I know the controlling rule and postion of joints in 3D via issus #106, I don't know how to calculate orientation for Blender load_pose, specially the rotation. Luckily, the code above saved me. This is my result:

Input Output 1 Output 2 I captured both output poses in the default front view. In the first output, the character is complety upside down. I manually set the global_orient paramenter to [[0,0,0]] and received the second output.

Hello, I want to know if both of these code modifications are in the fit_single_frame.py file? After following your approach for modifications, I encountered various errors including issues with tensors and arrays, among others. I suspect I might not have fully understood how you modified this piece of code. I noticed that in your modifications, there are two different lines: body_pose = vposer.decode(pose_embedding, output_type='aa').detach().cpu().numpy().reshape((1,63)) and body_model_forward = body_model.forward(body_pose=vposer.decode(pose_embedding).get('pose_body') if use_vposer else None). These lines shouldn’t coexist, right? Could you please explain more clearly how you modified this code? I appreciate your help and eagerly await your response.