some frame got bad performance and the loss just go around 0.04

RaineyWu commented 6 years ago

Expected behavior

i extract face image from video_A and video_B which is 1 mins or so video. and use the faceswap to train the A to B model. some frame learned very good, main feature is captured and the roi edges is smoothed and human eyes can not find some weak points. but some frames are just performed very bad that features does not learned in a acceptable way and the roi edges can easily tell. for the DA_loss when it is going down to 0.4 there is nothing reduce in the learning process.(which i think mean the model did not learn much). is there other way to get the model continue learn?

Steps to reproduce

i roughly use the default config. and setting the model to 128x128, and the database for A and B is around 470 images each. and use 1080Ti GPU trained for 1+ days.

Other relevant information

Command lined used (if not specified in steps to reproduce): faceswap.py ...
Operating system and version: Linux
Python version: 3.5
Faceswap version: commit hash or version number
Faceswap method: GPU 1080T
Other related issues:
... (for example, installed packages that you can see with pip freeze)

RaineyWu commented 6 years ago

sorry there is a small figure error in my issue. the DA_loss is 0.04

xielongze commented 5 years ago

I've encountered almost the same problem with you. I think is is with the weight, but don't know how to deal with it. Anyone help?

bryanlyon commented 5 years ago

This issue is extremely vague. Can you give us more information including specifics?

However, there are many reasons that the model may fail to learn. One is if there were extreme poses or expressions in just one of the datasets. Another is if there was a failure in the alignment for an image. There are more edge cases that could cause a failure to train.

xielongze commented 5 years ago

This issue is extremely vague. Can you give us more information including specifics?

However, there are many reasons that the model may fail to learn. One is if there were extreme poses or expressions in just one of the datasets. Another is if there was a failure in the alignment for an image. There are more edge cases that could cause a failure to train.

I'm willing to provide more information about it. But I'm new to GANs, so I have no idea what information is necessary or helpful. Can you give me a hint? Thank you.

bryanlyon commented 5 years ago

Some information that can help includes what kind of data you're using for training, is it video or photos? What model are you using? GAN is not recommended as it is incomplete Original or Original High Res will give much better results. Sample images are also very helpful in being able to see what the problem is.

Reviewing your information you posted, I see a few potential issues. First 470 images may be enough if properly selected, but are probably too few. We usually recommend 1000 images from a variety of angles, expressions, and lighting. Second, 1 day on a 1080ti (especially with so few images) is probably overtrained, this will suffer on any images that you didn't use to train, and would be a poor convert in general.

In general, I suggest you join us at our discord server https://discord.gg/FC54sYg where you can post samples and we can give you more help.

xielongze commented 5 years ago

Thanks. In my case it is with the training data. After I delete images with mic & hair in hand, the loss descends to about 0.031 in about 50k rounds. However, it descends in a very low speed now. Is it how it works?

bryanlyon commented 5 years ago

Loss is a logarithmic scale. As it gets closer to the ideal, it will go slower and slower. This is normal. Depending on the model and quality you're looking for it could take anywhere from 12 to 24 hours to maximize the quality. Loss is a poor measure for quality though, looking at the previews can give you a better idea of how good the swaps are.

Proper data management is key. Right now we don't have anything to handle obstructions like microphones, but that is something we're working on. Removing poorly performing images is a good idea in general as the model can struggle to identify the important features. Now, if the obstructions are on the video you want to swap to it's more complicated. That may require other techniques instead of just deleting the offending image.

deepfakes / faceswap-playground