Talking Face with just audio input

[bycloudai]

Hi, thank you for your amazing work.

I am just wondering if it's possible to render without a mouth frame and just based on the audio?(similar to what Wav2Lip does)

If so, can you tell me how to do it? Because I've been trying to figure it out if it's possible and keep on running into Exception: None Image error if I put the paths for mouth frames to None and amount of frames to 0 in demo.csv

[Hangz-nju-cuhk]

Hi, of course you can. Sorry that I have made a mistake in the dataloader and I think I have fixed it. Please pull the code and try again!

[bycloudai]

Hi, thank you for your fix. I can now run with an input image, an audio source and a pose source.

demo.csv -> misc/Input/image 1 misc/Pose_Source/video 120 misc/Audio_Source/voice.mp3 None 0 None

However, I still cannot put pose source as None and 0, and just have an input image and an audio source.

demo.csv -> misc/Input/image 1 None 0 misc/Audio_Source/voice.mp3 None 0 None

It produces the same error Exception: None Image

[bycloudai]

by the way, the YouTube video on your project page doesn't work.

[Hangz-nju-cuhk]

@bycloudai I have tested with pose source as None and 0 and it works fine. Please check again.

Meanwhile, my code will actually save this result even with any other pose source if you erase the --driving_pose flag in demo_vox.sh. You can also simply change the pose source path to be the input folder path.

[bycloudai]

@Hangz-nju-cuhk Do you mind sharing what you have inside demo.csv? I removed the --driving_pose flag for all the following tests. I pulled the latest code twice too just to double check, and I tried misc/Input/face 1 None 0 misc/Audio_Source/voice.mp3 None 0 None and its still not working. The error is Exception: None Image I get the same error if I try misc/Input/face 1 misc/Input/face 1 misc/Audio_Source/voice.mp3 None 0 None

There is exactly 1 image in the face folder

Btw, I'm running it on Windows, if that matters.

[Hangz-nju-cuhk]

I am just running misc/Input/517600055 1 None 0 misc/Audio_Source/681600002.mp3 None 0 dummy and it works fine. You can look into the voxtext_dataset.py and check the content of self.pose_frame_path when getting items. The logic of my code is actually quite easy.

[bycloudai]

Thank you so much for your patience. It works now! It worked without removing the --driving_pose flag in demo_vox.sh and have misc/Input/face 1 None 0 misc/Audio_Source/voice.mp3 None 0 dummy in my demo.csv