search

tanshuai0219 / EDTalk

[ECCV 2024 Oral] EDTalk - Official PyTorch Implementation
Apache License 2.0
329 stars 31 forks source link

How to lipsync an video with other audio? #37

Open linhcentrio opened 2 weeks ago

linhcentrio commented 2 weeks ago

Can you manual How to lipsync an video with other audio?

linhcentrio commented 1 week ago

https://github.com/user-attachments/assets/e39eb9e3-6ffe-4e8d-b5f2-39ba8264e04f python demo_change_a_video_lip.py --source_path test_data/green_screen_video.mp4 --audio_driving_path test_data/mouth_source.wav --save_path res/green_screen_video.mp4

https://github.com/user-attachments/assets/37f9c729-6957-42ad-ac72-0bb371ee42c1 python data_preprocess/crop_video.py --inp test_data/green_screen_video.mp4 --outp test_data/green_screen_video_crop.mp4 python demo_change_a_video_lip.py --source_path test_data/green_screen_video_crop.mp4 --audio_driving_path test_data/mouth_source.wav --save_path res/green_screen_video_crop.mp4

Why after crop_video result so bad?

tanshuai0219 commented 1 week ago

https://github.com/user-attachments/assets/e39eb9e3-6ffe-4e8d-b5f2-39ba8264e04f python demo_change_a_video_lip.py --source_path test_data/green_screen_video.mp4 --audio_driving_path test_data/mouth_source.wav --save_path res/green_screen_video.mp4

https://github.com/user-attachments/assets/37f9c729-6957-42ad-ac72-0bb371ee42c1 python data_preprocess/crop_video.py --inp test_data/green_screen_video.mp4 --outp test_data/green_screen_video_crop.mp4 python demo_change_a_video_lip.py --source_path test_data/green_screen_video_crop.mp4 --audio_driving_path test_data/mouth_source.wav --save_path res/green_screen_video_crop.mp4

Why after crop_video result so bad?

which one is your source video?
Assuming that test_data/green_screen_video.mp4 is the source video, the correct process should be:

  1. python data_preprocess/crop_video.py --inp test_data/green_screen_video.mp4 --outp test_data/green_screen_video_crop.mp4
  2. Make sure the fps of test_data/green_screen_video.mp4 is 25fps using ffmpeg -i test_data/green_screen_video_crop.mp4 -r 25 test_data/green_screen_video_fps25.mp4
  3. python demo_change_a_video_lip.py --source_path test_data/green_screen_video_fps25.mp4 --audio_driving_path test_data/mouth_source.wav --save_path res/green_screen_video.mp4

It should work.