search

ShunyuYao / DFA-NeRF

MIT License
72 stars 4 forks source link

The generated video has a stationary mouth and blurs the video. #3

Closed westernbuptboy closed 1 year ago

westernbuptboy commented 1 year ago

Thank you very much for disclosing your source code. After seeing the introduction video of your project, I think the effect is excellent. However, I encountered a slight issue during the process of running the source code. The video generated using the test script could not have its mouth closed, and there were obvious quality issues with the video. Additionally, the generated video has no sound, as shown in the following figure:

https://github.com/ShunyuYao/DFA-NeRF/assets/41902297/43ef207f-2341-4f1c-b548-d0abd86030e6

I set it when running the test script --resume dataset/train_together/obama_TrainExpLater_smoMix/600000.tar

ShunyuYao commented 1 year ago

Thank you very much for your interest in our project and for bringing this issue to our attention. I have already started to investigate the problems you have mentioned. Once I have determined the cause of these problems, I will correct it and tell you as soon as possible.

westernbuptboy commented 1 year ago

Thank you very much for your interest in our project and for bringing this issue to our attention. I have already started to investigate the problems you have mentioned. Once I have determined the cause of these problems, I will correct it and tell you as soon as possible.

Thank you very much for your reply. I have resolved this issue, which was caused by the deep speed engine not being fully downloaded. I downloaded the deepspeed again and solved the problem. At present, the synchronization effect is quite good, but there is a question that I would like to discuss with you. Have you ever considered posture reconstruction, as most nerf based schemes only reconstruct the part above the shoulder. If you want to output effects that include hand or leg movements, you may need to reconstruct the limbs or concatenate them with the original video. I would like to discuss it with you.

ShunyuYao commented 1 year ago

I'm very glad that you have resolved the issue. The posture reconstruction is not my main interests. But there are some papers about reconstructing posture with NeRF you can refer to. Here are some of them:

  1. Hu, Shoukang, et al. "SHERF: Generalizable Human NeRF from a Single Image." arXiv preprint arXiv:2303.12791 (2023).
  2. Xu, Hongyi, Thiemo Alldieck, and Cristian Sminchisescu. "H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion." Advances in Neural Information Processing Systems 34 (2021): 14955-14966.
  3. Su, Shih-Yang, et al. "A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose." Advances in Neural Information Processing Systems 34 (2021): 12278-12291.
  4. Peng, Sida, et al. "Animatable neural radiance fields for modeling dynamic human bodies." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
ShunyuYao commented 1 year ago

I will close the issue because it has been resolved. You can contact me with my email if you want to discuss more details. Thank you.

alchemician commented 1 year ago

@westernbuptboy can you explain me how exactly did you resolve this issue? facing the exact same issue

ZiqiaoPeng commented 1 year ago

@westernbuptboy I have the same problem, could you describe in detail how you solved it?

westernbuptboy commented 1 year ago

@westernbuptboy can you explain me how exactly did you resolve this issue? facing the exact same issue

download the deepspeech to replace the exist one in this project

westernbuptboy commented 1 year ago

@westernbuptboy I have the same problem, could you describe in detail how you solved it?

download the deepspeech to replace the exist one in this project