williamyang1991 / Rerender_A_Video

[SIGGRAPH Asia 2023] Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation
https://www.mmlab-ntu.com/project/rerender/
Other
2.96k stars 202 forks source link

What Key frame frequency (K) suggested? 10 works but when I make 1 it causes error #93

Closed FurkanGozukara closed 1 year ago

FurkanGozukara commented 1 year ago

I still got an output made from keys i presume

but this command not working

python video_blend.py result\posing_1_frame --beg 1 --end 292 --itv 1 --key keys --output result\posing_1_frame\blend.mp4 --fps 30.0 --n_proc 2 -ps

here 10 keyframes no error output vs 1 error happened output

https://github.com/williamyang1991/Rerender_A_Video/assets/19240467/32dbb448-f2e3-4c4c-a819-5210e360b8d7

https://github.com/williamyang1991/Rerender_A_Video/assets/19240467/e47bee78-9dc2-42a5-b8ad-50efc430de69

williamyang1991 commented 1 year ago

Yes, this is where this code improves the huggingface version. Our method will accumulate error when running long videos. So we only use stable diffusion to generate key frames to avoid error accumulation, and use ebsynth to generate other frames. If you set K=1, then the error will be easily accumulated.

williamyang1991 commented 1 year ago

We usually use K=5~20.

We have also provide how to avoid error accumulation with the options: https://github.com/williamyang1991/Rerender_A_Video#option2

FurkanGozukara commented 1 year ago

We usually use K=5~20.

We have also provide how to avoid error accumulation with the options: https://github.com/williamyang1991/Rerender_A_Video#option2

  • Pixel-aware fusion strength: The strength to preserve the non-inpainting region. Small to avoid error accumulation. Large to avoid burry textures.
  • Pixel-aware fusion detail level: The strength to sharpen the inpainting region. Small to avoid error accumulation. Large to avoid burry textures.
  • Smooth fusion boundary: Check to smooth the inpainting boundary (avoid error accumulation).

thank you