About MPI-INF-3DHP data formats

[MooreManor]

@mks0601

https://github.com/mks0601/NeuralAnnot_RELEASE/blob/2d688802f48ff1fc7b755aa1927c13a4acabeb65/MPI-INF-3DHP/demo_smpl.py#L143 In the demo code, you use subject_idx, seq_idx, and frame_idx as indexes. I check the original MPI-INF-3DHP datasets, and it uses subject_idx, seq_idx, video_idx, and frame_idx. How to match the video_idx?

And is the num of NeuralAnnot MPI-INF-3DHP pseudo GT the same as the fits used in SPIN?

[mks0601]

Please give some link, which uses video_idx. Maybe you can count the number of samples by yourself. Thanks!

[MooreManor]

@mks0601 SPIN preprocessing script

Does NeuralAnnot MPI-INF-3DHP pseudo GT include all the dataset or just some of it?

[mks0601]

I guess the video files contain same frames but from different viewpoints? NeuralAnnot provides 3D pseudo-GTs of all MPI-INF-3DHP dataset, and viewpoint is not required as 3D pseudo-GTs are defined in the 3D space.

[MooreManor]

@mks0601 I check the data, and the data is from different viewpoints as you said. Thank you for the in-time reply!

[MooreManor]

@mks0601 I tried visualizing the mesh in root-relative coordinates in SPIN, but I found that there was something wrong with the rendered results. I didn't follow the NeuralAnnot demo_smpl.py, and only pose and shape params in NeuralAnnot were used since it was root-relative. I estimated the camera translation with this code snippet. The focal length is assumed to be 5000, res to be 224, and camera rotation to be an identity matrix. Why rendering in this way cannot get a correct result? Did I do something wrong?

The green one is rendered with NeuralAnnot using only pose and shape params in it.

[mks0601]

Estimate global rotation as well

[MooreManor]

@mks0601 I check the original code of SPIN, and it seems like both the pose param and joint_3d depend on the viewpoint. So maybe I should change the pose params from the world coordinate to the camera coordinate.

[mks0601]

That is why I said you should estimate global rotation as well

[MooreManor]

Okay. I will take it a try.

[MooreManor]

@mks0601 After I correct the global rotation, the results are correct now. Most of the NeuralAnnots are satisfying, but some of them are in weird shapes like below.

How to filter out these unsatisfying ones?

[mks0601]

Actually, there is no way to filter them as only available evidence is sparse set of 3D joint coordinates.. One might try NeuralANnot with stronger L2 regularizer, but it might produce high data term error. There are pros and cons

[MooreManor]

Thanks, it's helpful. From the visualized results, many of NeuralAnnots are better than SPIN offered.