search

Fictionarry / ER-NeRF

[ICCV'23] Efficient Region-Aware Neural Radiance Fields for High-Fidelity Talking Portrait Synthesis
https://fictionarry.github.io/ER-NeRF/
MIT License
1.07k stars 137 forks source link

The effect is excellent #42

Closed langzizhixin closed 1 year ago

langzizhixin commented 1 year ago

Great project, thank you to the author for their excellent work. The effect of my own training is also very good.

https://github.com/Fictionarry/ER-NeRF/assets/74521932/a271b93f-5bb0-46da-94eb-1f9c8db66712

https://github.com/Fictionarry/ER-NeRF/assets/74521932/7f9ec6fd-0e6b-4b76-b4b3-b712b86cf63c

langzizhixin commented 1 year ago

Joining the lower body together, combined with hand movements, results in an internationally leading level.

langzizhixin commented 1 year ago

450p and 512p

tailangjun commented 1 year ago

niubility

MoroseYu commented 1 year ago

Is it good to shake your head like this?

tailangjun commented 1 year ago

shake your head

The result of my experiment is that the head will shake

MoroseYu commented 1 year ago

The second video had such a severe head shaking that he claimed it was at an international advanced level, which was unbelievable.

Fictionarry commented 1 year ago

Great project, thank you to the author for their excellent work. The effect of my own training is also very good.

keke.mp4 final.14.mp4

Thanks for your appreciation :)

Is it good to shake your head like this?

The result of my experiment is that the head will shake

The second video had such a severe head shaking that he claimed it was at an international advanced level, which was unbelievable.

The shaking of the head part is a very common and simple problem, mainly due to the inconsistency of face tracking. Nevertheless, this step is not the main contribution of our work and we have no plan to improve it. Currently, you can try adding --smooth_path to relieve it. https://github.com/Fictionarry/ER-NeRF/issues/18

This repo is released for research purposes. If more delicate results are needed for commercial usage, please consider replacing the Basel 3DMM model and optimizing the whole pipeline.

DeepMakerAi commented 1 year ago

Joining the lower body together, combined with hand movements, results in an internationally leading level.

Hello, the effect you synthesized is very good. May I know which tool is used to connect the head and body

chaolongy commented 1 year ago

Great project, thank you to the author for their excellent work. The effect of my own training is also very good.

keke.mp4 final.14.mp4

Your result is well. I have one question is your audio feature extractor used a DeepSpeech, Wav2Vec or HuBERT?

Real-or-unreal commented 1 year ago

The first video exhibits a level of realism and detail that is not typically achievable through this work alone. It is more plausible that the first video was either the original source material or created using Tencent's Zenvideo technology, known for its impressive visual effects and realistic rendering, albeit with the alteration limited to the character's lips.

langzizhixin commented 1 year ago

--smooth_path

langzizhixin commented 1 year ago

第一个视频展示了一定程度的真实性和细节，通常仅通过这项工作无法实现。更有可能的是，第一个视频要么是原始素材，要么是使用腾讯的 Zenvideo 技术创建的，该技术以其令人印象深刻的视觉效果和逼真的渲染而闻名，尽管改动仅限于角色的嘴唇。

It is trained using Tencent's digital human and then inputting ER-NeRF. I am lazy to search for models to shoot videos.

langzizhixin commented 1 year ago

很棒的项目，感谢作者的出色工作。我自己训练的效果也很好。 咳咳.mp4 Final.14.mp4

感谢您的赞赏:)

这样摇头好吗？

我的实验结果是头会摇头

第二个视频摇头晃得厉害，他还声称达到了国际先进水平，这让人难以置信。

头部抖动是一个非常常见且简单的问题，主要是由于人脸跟踪的不一致造成的。然而，这一步并不是我们工作的主要贡献，我们也没有计划对其进行改进。目前可以尝试添加--smooth_path来缓解。#18

该存储库是出于研究目的而发布的。如果商业用途需要更细腻的结果，请考虑更换Basel 3DMM模型并优化整个流程。

Thank you for the author's reply. When shooting the original video, shaking your head slightly will result in better results. Plus the source code, the author has - smooth_ path, it's perfect.

Jacoo-ai commented 3 months ago

Great project, thank you to the author for their excellent work. The effect of my own training is also very good.

keke.mp4 final.14.mp4

Excellent work! I found the lip movement in the first video is very well. Did you change some parameters during training, or directly follow the intructions provided by the author?

gg22mm commented 1 month ago

Great project, thank you to the author for their excellent work. The effect of my own training is also very good.伟大的项目，感谢作者的出色工作。我自己训练的效果也很好。 keke.mp4 final.14.mp4

都执行了啥？为什么我没有这么好的效果？哪里出问题了？

//训练 python main.py data/me/ --workspace trial_me/ -O --iters 100000 #头部训练 python main.py data/me/ --workspace trial_me/ -O --iters 125000 --finetune_lips --patch_size 32 #微调嘴型动作 python main.py data/me/ --workspace trial_me_torso/ -O --torso --head_ckpt trial_me/checkpoints/ngp_ep0041.pth --iters 26 #训练身体 --iters 200000

训练是这三步吗？ 如果 拿一个没有说话的视频 来 训练会怎么样？到时能说话吗？