search

georgeretsi / smirk

Official Pytorch Implementation of SMIRK: 3D Facial Expressions through Analysis-by-Neural-Synthesis (CVPR 2024)
https://georgeretsi.github.io/smirk/
MIT License
126 stars 10 forks source link

How to reduce jitter in rendered results #6

Open ak01user opened 2 months ago

ak01user commented 2 months ago

Hi!

Amazing work!

Is there any smoothing operation?

filby89 commented 2 months ago

Hi ! In SMIRK we did not consider any kind of temporal adjustments or smoothing operations - the method was built to be applied in separate images.

akk-123 commented 1 month ago

@filby89 if we want use smirk in video, is there any way to improve the smoothness of the video?

georgeretsi commented 1 month ago

Hi there, there are two ways to tackle this: you can use a post-processing temporal filter that smooths the output of consecutive frames, or you can retrain the smirk pipeline with a temporal convolution using video data as done in our previous work Spectre (https://github.com/filby89/spectre).

Note that if you want to remove jittering in the image reconstruction space, a re-training for temporal consistency is needed, along with selecting a specific set of "anchor" face pixels during inference. This is something we plan to explore and resolve in the future.