Is this the right training process?

[Qingcsai]

Hi, shunsukesaito!

I am traing PIFu using my own datasets, which have 300 human mesh for traing, 100 mesh for testing. I set learning_rate=0.001, Sigma=0.1,batch_size=16, in the first 5 epochs, the error during training is shocking between 0.07-0.16, is seems like won't converge? When I check some mesh generated during testing at the end of each epoch, it seems weired, here's the result at epoch 4:

generated mesh:

the input image:

Do you think it the resonable results during training? How to converge fast? I hope I didn't bother you, expecting to your reply! thanks!

[shunsukesaito]

The error and reconstruction quality seems worse than what it should be. Although it's difficult to accurately diagnose with your specific scenario, there are several educated guesses:

• Somehow your reconstruction result is unnaturally symmetric along z axis. You may as well check if any image augmentation (e.g., flipping) results in inaccurate alignment between input image and project 3d points.

• Depending on your data scale, you need to adjust sigma value. You can dump out sampled point clouds as obj files or so and load them together with the original mesh in MeshLab to see the range of perturbation is not too large or too small.

Thanks for trying out PIFu framework!

[Qingcsai]

Thanks for your quickly reply! In fact, my original datasets before using render_data.py is not standing along Y axis, but the human standing along Z axis(facing the negative Y axis), which means the ground truth mesh has 90° difference from your provided free mesh.

So I did one thing during rendering, I changed R = np.matmul(R, make_rotate(math.radians(90),0,0)) with R = np.matmul(R, make_rotate(math.radians(-90),0,0)) in

https://github.com/shunsukesaito/PIFu/blob/30b428ba74bd7743a17c19fa20f6bfd39b1de057/apps/render_data.py#L218

Then I got the correct direction of the ground truth mesh when generating data.

Before changing:

After changing:

I wonder is it enough changes in codes for trainging shape? Or is it the reason why resulting in inaccurate alignment between input image and project 3d points?

Looking forward to your reply! Thanks.

[shunsukesaito]

I would suggest projecting the sampled points of the body mesh onto input images using the same code as your projection operator. If everything is correct, you should be able to see the projected points aligned with the input image,

[Qingcsai]

Okay, thanks for your advice, I would try with it.