Offset on SMPL

[MarilynKeller]

With the current code, the generated AMASS sequence has feet below zero and an offset compared to the mocap markers :

This is caused by the change of coordinate to get Y axis up : https://github.com/eth-ait/aitviewer/blob/4aac6fd77bab30674db5858e83e0ae64f0a8a7d4/aitviewer/renderables/smpl.py#L177

I think the reason is that the rotation is applied to the root_joint of SMPL. Yet the root joint of SMPL is not the origin of SMPL (this detail is a common source of error). If you display SMPL in Tpose, the root joint is typically 20cm below the world origin.

For now, I removed those lines in the .from_amass method and instead apply the rotation directly on the vertices in fk()

    def fk(self):
        """Get joints and/or vertices from the poses."""
        verts, joints = self.smpl_layer(poses_root=self.poses_root,
                                        poses_body=self.poses_body,
                                        poses_left_hand=self.poses_left_hand,
                                        poses_right_hand=self.poses_right_hand,
                                        betas=self.betas,
                                        trans=self.trans)
        skeleton = self.smpl_layer.skeletons()['body'].T
        faces = self.smpl_layer.bm.faces.astype(np.int64)
        joints = joints[:, :skeleton.shape[0]]

        #Transform output vertices and joints into our viewer's coordinate system (where Y is up).
        to_y_up = torch.Tensor([[1, 0, 0], [0, 0, 1], [0, -1, 0]]).to(self.device)
        verts = torch.matmul(to_y_up.unsqueeze(0), verts.unsqueeze(-1)).squeeze()
        joints = torch.matmul(to_y_up.unsqueeze(0), joints.unsqueeze(-1)).squeeze()

This fixes the offset issue:

[kaufManu]

Thanks for reporting this, we'll probably get this fixed soon. So actually is the origin of SMPL unknown then? That's actually been a question that I've had for a while :) I.e. is applying the transformation to the vertices directly the only solution?

[kaufManu]

Fixed with 8ed00d33ccd12449fece9714f072ed7d632795ec.

[MarilynKeller]

Let me try to detail, it took me some time to wrap my head around the thing after a while :).

Let's define: V: Transformed SMPL vertices T(\beta): Template vertices, with beta applied J_0 : Root joint location for unposed T(\beta) (Note that this is subject specific) R^(J_0): Root joint rotation matrix t : SMPL translation

SMPL applies the root joint rotation at the template origin (so apply a -J_0 translation, apply the rotation and then do +J_0) eq. (1). In order to rotate the result, we do eq. (2), which is equivalent to eg. (3).

In summary, if you want to do the rotation in load_from_amass, you will have to do eq. (4) and eq. (5). The only difference with what you had before is this J0. (And indeed the offset I observed had the value Ryup * J_0

To get J_0 in load_from_amass, you'll have to compute a SMPL forward pass to get the location of J0 in Tpose (V, J = SMPL(beta, pose=0) , which is a bit inconvenient because the class object is not initialized yet. It is indeed not very clean to have the rotation in fk() at the moment, on the other side, having it in load_from_amass requires more explanation and a forward pass, so up to you :).

[MarilynKeller]

And you can recover the location of the origin of SMPL at anytime by doing J, V = SMPL(beta, theta), smpl_origin = J[0] - trans.

[kaufManu]

Thank you for the detailed explanation! We've kept it in the fk function as you originally suggested to avoid a forward pass necessary to obtain the origin (because it's a bit easier to explain 😄).

The fix is included in the latest version (1.1, pip install --upgrade aitviewer).