search

Rudrabha / Wav2Lip

This repository contains the codes of "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild", published at ACM Multimedia 2020. For HD commercial model, please try out Sync Labs
https://synclabs.so
10.65k stars 2.27k forks source link

How to have program ignore faceless frames without excluding them or throwing an error? #190

Closed MalachiMassey closed 3 years ago

MalachiMassey commented 3 years ago

The title is pretty explanatory. I just want to export videos where the frames with faces have the lip movement changed, but the faceless frames are ignored, but still included as normal in the final product. This can be helpful in the case of fixing dubbed videos where the speaker/actor briefly leaves the frame.

prajwalkr commented 3 years ago

You need to exclude that frame from being passed to Wav2Lip, and generate results for the rest of the frames. For the missing frames, simply use the original video frame without any changes.

MalachiMassey commented 3 years ago

That seems rather labor-intensive in the case that an actor keeps ducking out of a frame or something similar. Is there are way to have these frames ignored by editing the code or run parameters?

Rudrabha commented 3 years ago

You can change things inside this if statement . Instead of throwing this error, you can appropriately handle it. For example you can store the frame (or even the frame number) in a separate dictionary. You can further give some dummy (x1, y1), (x2, y2) in these cases.. While generating the final video, you need to use the original frame for all the frame numbers present in the undetected array. Hope this gives you an idea on what to do. Please also submit a pull request if you are able to handle such cases. We will like to add it as an option in our code.

MalachiMassey commented 3 years ago

Thank you. I will look into it and see what I can come up with.

prajwalkr commented 3 years ago

Please re-open again if required

dheerajmpai commented 3 years ago

A small trick we had used was to just run wav2lip on the first pixel when there is no face. This works fine with us.

MalachiMassey commented 3 years ago

What do you replace the error throwing code with in that case, @dheerajmpai? I suppose this is more of a work-around than a viable feature, but I'd like to at least give it a try.

SaadBazaz commented 2 years ago

@dheerajmpai Where did you use this code? Would need this too.

SaadBazaz commented 2 years ago

A switch like --use-original-where-faceless-frames or something would be great.