search

ZiqiaoPeng / SyncTalk

[CVPR 2024] This is the official source for our paper "SyncTalk: The Devil is in the Synchronization for Talking Head Synthesis"
https://ziqiaopeng.github.io/synctalk/
Other
1.33k stars 161 forks source link

Questions About Enhancing Video Quality #78

Open schxnhxlz opened 6 months ago

schxnhxlz commented 6 months ago

First off, I want to say thanks for putting together such a cool project! I've really enjoyed playing around with it and have learned a ton.

I've got a couple of questions about improving the video output quality and was hoping you could help me out:

Higher Resolutions: Is it possible to generate videos at resolutions higher than 512x512 to improve the visual quality? If so, how can I adjust the settings to enable higher resolutions?

Bitrate Adjustment: The output video bitrate seems a bit low and might be impacting the quality. Could you point me to where in the code or config files I can tweak the bitrate settings for video encoding?

Thanks again for all your hard work!

Cheers!

HenryKang1 commented 6 months ago

I am not the author however what I would like to recommend the Apply the SR algorithm for this. The input video is cropped face basically. After that blend with your original video. This can improve your video.

schxnhxlz commented 6 months ago

I am not the author however what I would like to recommend the Apply the SR algorithm for this. The input video is cropped face basically. After that blend with your original video. This can improve your video.

hi @HenryKang1 thanks for your reply. what is the sr algorithm? I already have a ffmpeg command which puts the output video onto the raw video.

Inferencer commented 6 months ago

He's referring to a super resolution model such as codeformer or gfpgan which would upscale the face at some cost to skin fidelity although I don't see one implemented in this repo at first glance I could be wrong.

HenryKang1 commented 6 months ago

He's referring to a super resolution model such as codeformer or gfpgan which would upscale the face at some cost to skin fidelity although I don't see one implemented in this repo at first glance I could be wrong.

It is correct. Super resolution and pyramid blending or other type of blending is common for computer vision and video processing. FFmpeg only help very low level but it is not gonna work for this problem.

schxnhxlz commented 6 months ago

Thank you for your answer. I find that upscaling can quickly look fake. I would prefer to be able to adjust the quality during training.

wning13 commented 6 months ago

You can try using Topaz Video AI to enhance video quality. However, if you want to directly improve the output quality of the model, you'll need to adjust the model's structure and require more computational resources.

schxnhxlz commented 6 months ago

That one helped: https://github.com/ZiqiaoPeng/SyncTalk/issues/102#issuecomment-2106812257