How to run video-based reenactment demo

[xiao-keeplearning]

Thank your excellent work! How to use a source image and a drive video to generate video-based reenactment？The demo you provided is image-based head reenactment.

[YuDeng]

Hi, you can simply use consecutive frames from a video clip as the driving images. Our method extracts per-frame motion embedding from the images for reenactment.

[johndpope]

fyi - this code wraps up this nicely https://github.com/Zejun-Yang/AniPortrait python -m scripts.vid2vid --config ./configs/prompts/animation_facereenac.yaml -W 512 -H 512 -acc

https://github.com/Zejun-Yang/AniPortrait/blob/main/scripts/vid2vid.py

[xiao-keeplearning]

@YuDeng I follow the commands in Data Preprocessing for Custom Images section and got the following result, which didn't work well. The face is shaking abnormally，and face fluctuates in size. https://github.com/YuDeng/Portrait-4D/assets/26853334/5ba96e4d-34ec-4158-a69e-873185131876 And, I noticed that there are use_smooth parameters in the image preprocessing code, does this enable it improve the result? Do you have any suggestions on how to reproduce what you showed in the demo?

[YuDeng]

@YuDeng I follow the commands in Data Preprocessing for Custom Images section and got the following result, which didn't work well. The face is shaking abnormally，and face fluctuates in size. https://github.com/YuDeng/Portrait-4D/assets/26853334/5ba96e4d-34ec-4158-a69e-873185131876 And, I noticed that there are use_smooth parameters in the image preprocessing code, does this enable it improve the result? Do you have any suggestions on how to reproduce what you showed in the demo?

Set use_smooth=True in https://github.com/YuDeng/Portrait-4D/blob/4ef83925268faf5469a95545017e41a7a13b1090/data_preprocess/lib/inferencer.py#L93 can reduce the face jittering problem to some extent.

Furthermore, you can adjust the smoothing level by adjusting the hyper-parameters of the smoother: https://github.com/YuDeng/Portrait-4D/blob/4ef83925268faf5469a95545017e41a7a13b1090/data_preprocess/cropping/crop_images_portrait_model.py#L10.

[xiao-keeplearning]

@YuDeng Hi, I set use_smooth=True and use original the smoothing level , face jittering problem doesn't look relieved.

[YuDeng]

@YuDeng Hi, I set use_smooth=True and use original the smoothing level , face jittering problem doesn't look relieved.

Does this also happen with the final synthesis results of Portrait4D? It should be able to tolerate a certain degree of face jittering in the driving frames.

[xiao-keeplearning]

The final synthesis results is also face jittering

[YuDeng]

Can you provide a specific example?

[xiao-keeplearning]

drive video

https://github.com/YuDeng/Portrait-4D/assets/26853334/7a4a4529-d3ad-4f09-9023-9ecc38420815

synthesis result:

https://github.com/YuDeng/Portrait-4D/assets/26853334/f6146a26-dfe7-4c33-beb6-b5bf865d9b0d

I've found that in the synthesis video, the face gets bigger every time the face blinks.

[YuDeng]

Well, we also observe similar problem in our experiments and it can happen for certain subjects and driving frames. This is mainly due to the fluctuation of reconstructed camera extrinsics for training. The currently provided checkpoint is trained on our original pre-processed data which may inherit this issue. We've re-trained our model on better pre-processed data and we will try to provide the updated checkpoint in the future.