search

shaoanlu / faceswap-GAN

A denoising autoencoder + adversarial losses and attention mechanisms for face swapping.
3.36k stars 844 forks source link

Alignment Problem / 'Zoom' Problem? #67

Open petergerten opened 6 years ago

petergerten commented 6 years ago

Hi,

https://gfycat.com/LegitimateGlassImago

I used the 128px v2 GAN to produce this video without alignment. Can someone comment on what is going on here? Is this an alignment problem? (as the 128px GAN does not have a video conversion function with alignment yet)

Or is this this 'zoom-in' issue discussed on this and on the faceswap repo that seems to happen with the GAN model?

Peter

room3dev commented 6 years ago

This alignment problem not from the model. In fact, we don't have face alignment for GAN at this time so we don't have face matrix data for using cv2.warpAffine( ... ) to crop face precisely like we do in original deepfakes.

petergerten commented 6 years ago

@room3dev Thanks. So you agree what is visible in the video is an alignment (or lack thereoff) issue - right? I don't quite understand why the face matrix data would not be available with the GAN. Isn't that done by an external framework (face_recognition or face-alignment)?

room3dev commented 6 years ago

As far as I know, original deepfakes or FakeApp use 1adrianb/face-alignment to obtain 2d face landmark and covert it to transformation matrix then use this data to extract the face from an input image. You can find the code I mention here, https://github.com/dfaker/df/blob/master/align_images_masked.py