search

deepfakes / faceswap-playground

User dedicated repo for the faceswap project
306 stars 194 forks source link

Crash at the start of training when extra faces not present in alignment file #276

Closed Kirin-kun closed 5 years ago

Kirin-kun commented 5 years ago

Quite easy to understand.

When using a masked model or warp to landmarks, which uses the alignment files, if in a directory containing the faceset there are files not referenced in the file, it immediately crashes, saying it didn't find that face in the alignment file.

After extracting, I had added a few face from another extraction which had interesting poses and boom, crash.

Not really a blocking bug, but it at least should ignore such faces, not crash.

And it all the more makes me petition to do away with the alignments file and do it the Dfl way: embed the landmarks in the png files.

I think it would even be possible to have it both ways. The overhead at extraction shouldn't be that high.

torzdf commented 5 years ago

Happy for @bryanlyon @kvrooman to weigh in on this. Adding binary data to a png is non-standard. It also solves one issue whilst adding others, not least limiting target filetypes, and effectively breaking manual alignments tool.

In an ideal world (imho) Facesets would become a thing of the past. This is just my own personal workflow, not a project direction, but theoretically, if the tools were developed correctly, a video + an alignments file could be all you need at every step (with faces lifted straight from video files for training).

I don't see a perfect solution to this at present (and trust me, it has been discussed extensively on our Discord server). The alignments tool does try to fill this gap a little by allowing you to merge multiple alignments files into one, which is very useful for creating training sets.

The 'alignments file' problem has been on our list for a while, but the current dev team do not have a decent enough solution to implement at this moment in time.

Kirin-kun commented 5 years ago

You ideal world won't exist until there's a 100% accurate detector, with no false-positive. As of today, it's called a human.

That's one of the thing I appreciated in DeepFaceLab: you can extract from multiple sources, dump the interesting faces in a directory, and start training right away. And even if you rename the files, it will still work.

Embedding can be done both in png (using text data) and jpg (using exif) AFAIK, I've yet to see another format used in this project.

The way I see it, embedding the landmarks in the files could be used as a failover if they are not found in the alignment file.

That way, it won't break the existing workflow.

Kirin-kun commented 5 years ago

Not blocking, so I will just close it and see later how it will be handled.