Why are the 2D keypoints used?

[Saafke]

Hi,

Great paper and thanks for releasing the code!

I just had some questions. I can see in the code that the 2D keypoints (predicted by the STAF method) are used for an optional step Temporal SMPLify.

1. What is Temporal SMPLify exactly; I don't see this in the paper?

2. Why is it called Temporal SMPLify? I thought SMPLify works frame-by-frame. Perhaps because STAF uses temporal information to predict 2D keypoints (which are then used for SMPLify)?

3. Would it be possible/easy to change the code to use the default OpenPose keypoints (that includes more keypoints such as hands/feet/face, in comparison to STAF) to fit the SMPL-X body shape model in the Temporal SMPLify step in VIBE. Namely, I would like more accurate 3D feet estimations, which is provided by OpenPose+SMPL-X.

Thanks

[MichaelJBlack]

Temporal SMPLify is described in this paper: https://ps.is.tuebingen.mpg.de/publications/muvs-3dv-2017 and the code is here: https://github.com/YinghaoHuang91/MuVS

[John-HarringtonNZ]

Hi Michael, I agree that this is a great paper and thank you for releasing the code. I agree with @Saafke, I don't see a reference to Temporal SMPLIify anywhere. The links you provided refer explicitly to multi-view videos, when VIBE takes monocular input.

Is there some usage of the temporal strategy from that referred project in this code, or am I misunderstanding your explanation?

Thank you again!

[MichaelJBlack]

The multi-view model has two stage, the second of which uses a generic DCT temporal prior (see temporal_fit.py). You should be able to run the code with a single image stream and exploit the temporal prior or adapt this temporal prior to use it with any other version of SMPLify.

[John-HarringtonNZ]

Understood, thank you!

[MichaelJBlack]

You're welcome!