6d pose representation

[smandava98]

Hi,

I have one more question:

If you are converting to 6d poses and you are converting it back to rotation matrices how do you ensure that the final matrices are still valid? As far as I know, wouldnt this affect how it is displayed in the image?

[statho]

Please have a look at this function. It converts the 6D poses to a valid rotation matrices.

[smandava98]

I tried that however, if I call that function it doesn't return the same matrices. When I visualize on the image, it is way off @statho

Would converting it to axis angle then back to rotation matrices work better? I'm not sure how to go about this as the SMPL reconstruction on the image becomes way off.

[statho]

Just to make sure. a) Are you using the prepare_smpl_params function that internally converts the 6D poses to rotation matrices before running the SMPL model? b) Are you rotating the x-axis by 180 degrees before rendering, as done here?

[smandava98]

I am using this to turn my HMR2 predictions (which are in rotmat form) to 6d poses to train your diffusion network:

def matrix_to_rotation_6d(matrix: torch.Tensor) -> torch.Tensor: """ Converts rotation matrices to 6D rotation representation by Zhou et al. [1] by dropping the last row. Note that 6D representation is not unique. Args: matrix: batch of rotation matrices of size (, 3, 3) Returns: 6D rotation representation, of size (, 6) [1] Zhou, Y., Barnes, C., Lu, J., Yang, J., & Li, H. On the Continuity of Rotation Representations in Neural Networks. IEEE Conference on Computer Vision and Pattern Recognition, 2019. Retrieved from http://arxiv.org/abs/1812.07035 """ batch_dim = matrix.size()[:-2] return matrix[..., :2, :].clone().reshape(batch_dim + (6,))

I do this for body pose (not camera).

A) after running through the diffusion network, I use the rot6d_to_rotmat function and correctly access the right indices as you’ve done in prepare_smpl_params to get the rotation matrices for global orient and pose. Betas were never modified.

B) Yes, I am doing this

Thanks @statho

On Tue, Mar 19, 2024 at 1:32 PM Anastasis Stathopoulos < @.***> wrote:

Just to make sure. a) Are you using the prepare_smpl_params https://github.com/statho/ScoreHMR/blob/master/score_hmr/utils/utils.py#L75 function that internally converts the 6D poses to rotation matrices before running the SMPL model? b) Are rotating the x-axis by 180 degrees before rendering, as done here https://github.com/statho/ScoreHMR/blob/master/score_hmr/utils/mesh_renderer.py#L227 ?

— Reply to this email directly, view it on GitHub https://github.com/statho/ScoreHMR/issues/6#issuecomment-2008078669, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFNOJPZODGLYJG4XN7ZOJHLYZCOHRAVCNFSM6AAAAABE452EDSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMBYGA3TQNRWHE . You are receiving this because you authored the thread.Message ID: @.***>

[smandava98]

Also, followup:

Do you convert HMR's 3D camera parameters to anything? I see they are of shape 3 so I assume they just represent x,y,z in 3D space. Unsure if this needs to be converted to something or not.

[statho]

Please have a look at how we convert to 6D pose here. You will only need to omit the aa_to_rotmat() function call, if rotation matrices are used for the pose.

Please convert the rotation matrices to 6D, as we do in the highlighted code.