search

yuval-alaluf / stylegan3-editing

Official Implementation of "Third Time's the Charm? Image and Video Editing with StyleGAN3" (AIM ECCVW 2022) https://arxiv.org/abs/2201.13433
https://yuval-alaluf.github.io/stylegan3-editing/
MIT License
654 stars 73 forks source link

Match motion of smooth video with raw video #42

Closed ikcla closed 2 years ago

ikcla commented 2 years ago

Comparing smooth video with raw video, it shows obviously like a slow motion compare to original version. Is there away to match the motion of smooth video with original one? Is it possible to change motion in smooth video here?

average fine layers

result_latents[:, 9:, :] = result_latents[:, 9:, :].mean(axis=0)

change 9 to another number to change motion appearance?

Thanks.

yuval-alaluf commented 2 years ago

You are correct that there is some mismatch in the smooth video. This smoothing is actually caused not by the line you pasted above, but the following function: https://github.com/yuval-alaluf/stylegan3-editing/blob/ab01a5d90b8ba67e0da0e1388f0931482601006c/inversion/video/post_processing.py#L49-L52 Here, we replace each latent with the average of its neighboring latents. This causes the video to be smoother, but it does lead to a slight mismatch as a result. One way to help avoid this is to change the weights we used there, or come up with a different way to smooth the results. We offered a single approach for doing so, but there could be better ways to do so.

ikcla commented 2 years ago

@yuval-alaluf Thank you for your information. I am not familiar with this moving average mathematic in this case, but is there a quick short cut to change math to dedicated only on moving average for rotation to stable the head boundary only so that we can keep facial expression comparable with original version? Thanks.

ikcla commented 2 years ago

This topic could be another development project that apply unsupervise smooth AI algorithm.