subject-specific hand shapes (ß)

[wetoo-cando]

Hello @ychao-nvidia ,

In the paper it says "We pre-calibrate the hand shape β for each subject and fix it throughout each subject’s sequences."

Can you please shed some light on how this was done exactly?

Thanks.

[ychao-nvidia]

We built an interactive user interface for calibrating MANO shape parameter per subject. Below is a summary of the pipeline:

• We use 3 RealSense cameras, instead of 8 in capturing hand-object interaction.
• To get higher quality point clouds, we place these cameras closer to each other compared to the hand-object interaction case.
• The pipeline consist of two stages: (1) data collection, and (2) solving betas.
• In stage (1), we let the subject move their hand by moving, rotating, and articulating the fingers, in a moderate speed in front of the cameras. We only do the right hand and use the solved shape for the left hand.
• We track the hand pose in real time using the same Eq. 1. in the paper, except dropping the keypoint term $E\text{kpt}$ (i.e., only using the SDF term $E\text{depth}$ and regularization term $E_\text{reg}$). We fix the shape of MANO model to be the mean shape (i.e., betas is all zeros) for the real-time tracking here.
• The point cloud and the tracked pose are visualized on a screen and shown to the subject in real time.
• Meanwhile, we log the point cloud and the tracked hand pose at each frame to a data buffer.
• We end stage (1) after logging certain mount of data.
• In stage (2), we refine both hand pose and shape in full-batch optimization using the data logged in the buffer.
• We add an additional regularization term $E_\text{betas}$ to the energy function from stage (1) for additionally optimizing betas.
• We alternate between optimizing pose and shape parameter. I.e., optimize pose for $X$ iterations while fixing shape, and optimize shape for $X$ iterations while fixing pose, and repeat this process for $Y$ iterations.
• Finally, we save the solved shape parameter betas for that subject.

[wetoo-cando]

Thanks for the detailed response @ychao-nvidia !