search

facebookresearch / pifuhd

High-Resolution 3D Human Digitization from A Single Image.
Other
9.5k stars 1.44k forks source link

the training results are worse than pifu #70

Closed SFM2020 closed 3 years ago

SFM2020 commented 3 years ago

I trained pifuHD according to the description of your paper with muti-view，but the results are not good as expected.Is there anyting wrong？ image

zener90818 commented 3 years ago

Hi，Could you please share the training code with me，many thanks.

shunsukesaito commented 3 years ago

It'd be difficult to diagnose issues by only checking results, but there could be a couple of steps to figure out which part is causing deterioration.

  1. Have you successfully trained a multi-view model with PIFu not PIFuHD? If it's not working well, likely the training code for multi-view integration has some issues.
  2. Have you successfully trained a single-view PIFuHD? If not, this might be caused by discrepancy in the training code.
SFM2020 commented 3 years ago

I have trained multi-view model with PIFu , it's working well, but I don't train a single-view PIFuHD.I sent you an email with more details, but no one replied,maybe the email address shunsuke.satio16@gmalil.com is wrong.

shunsukesaito commented 3 years ago

The last name of the email is not correct (also it should be gmail.com). You can resent an email.

SFM2020 commented 3 years ago

I'm very sorry, shunsuke.saito16@gmail.com is the email address I sent，Shall I send an email to this address again？

shunsukesaito commented 3 years ago

Yes you can. But why don't you post more details here so that others can also benefit?

SFM2020 commented 3 years ago

Good ieda！ I trained 3 view model with pifuHD ,the code is modified from pifu of 3 view,which is working well.The changing detials are as follows:

  1. During training, the fine level random crop 3 view images from 1024 to 512 only once in each iteration, and the images with a size of 1024 used for evaluation.
  2. No normal training is added. Point sampling strategy is the same as your paper.
  3. Fine-tune from your pre-training pifuhd model.
  4. Using camera model of perspective projection which is working well in pifu of 3 views.
  5. gen_mesh() is the same as pifu , unlike your PifuHD

Figure 1 below is the result of my training with one batch. The training set is the same as the validation set. But the strange thing is that during training,the IOU quickly increased to 0.8, it began to decline, and the reconstructed mesh became worse and worse,as shown in Figure 2,.What's more,the training loss has been declining, and the test loss first decreased and then increased in the same dataset. image

Apokar commented 3 years ago

@SFM2020 加个QQ吗?有些问题想请教下:729975476

shunsukesaito commented 3 years ago

Sorry for late response. What you do sounds reasonable to me. One thing is in the multi-view case, cropping of 3 views needs to make sure sampled points are within the cropping of all views. If any views are missing, it might result in unexpected behavior.

SFM2020 commented 3 years ago

Sorry for late response. What you do sounds reasonable to me. One thing is in the multi-view case, cropping of 3 views needs to make sure sampled points are within the cropping of all views. If any views are missing, it might result in unexpected behavior.

In my opinion, pifuhd with multi-view cannot be trained well under the strategy of cropping.After cropping images from different views in the same way, there is no guarantee that the sampling points can be seen from different views at the same time. Am I right？And Did you implement multi-view PifuHD?

shunsukesaito commented 3 years ago

Sorry for late response. What you do sounds reasonable to me. One thing is in the multi-view case, cropping of 3 views needs to make sure sampled points are within the cropping of all views. If any views are missing, it might result in unexpected behavior.

In my opinion, pifuhd with multi-view cannot be trained well under the strategy of cropping.After cropping images from different views in the same way, there is no guarantee that the sampling points can be seen from different views at the same time. Am I right？And Did you implement multi-view PifuHD?

I think your understanding is correct. In fact, you have to carefully choose views and points such that these points can be seen from multiple view points. I have not trained multi-view PIFuHD so it might need extra work to get it working.

SFM2020 commented 3 years ago

Many thanks for your kind help !

ChanYeongShin commented 3 years ago

@SFM2020 Can you share the modified training code for multi-view setup with PIFu?