Tollanador
commented
2 years ago DeepFaceLab has it's own Encoder and Decoder modules.
StyleGan3 outputs a 2D image.
If I understand DeepfAceLab enough,
Are you asking if you fine-tune StyleGan3 with a dataset of 1 single person from many different angles and lighting conditions, different facial expressions, etc, can you then use that trained model ( ~250 mbish file) with DeepFaceLab or similar?
You would also need to do the same with the original face, so that you have 2 models. The Target model, and the Original face model.
The answer, with a lot of work, yes, probably.
You would have to add the StyleGan3 model to DeepFaceLab's models. You pretty much need to use the same deep learning architecture with the model weights from that architecture . The weights expect to be put into the same layer types and size. They are kind of meaningless without this.
Before you go diving into all that you should probably consider how StyleGan3 generates a final image, particularly in the upper layers. These upper layers tend to 'paint' the most fine-grained details onto the image, such as hair. What gives it such a realistic output, is a level randomness used in that generation. This is perfectly fine for a single image output.
When you try to use those outputs, strung together to create a video, you will see those fine-grained details flicker around. The exact structure of each hair for example, will change from frame to frame. The overall structure of the face and hairstyle will remain consistent enough though, I think.
As such, the current StyleGan3 model, without modifications, is probably not suitable for DeepFaceLab applications, or at least terribly inefficient.
sabotage3d
Once we have trained the model and created a single generated face, is it possible to export it as a new model? For example for using it in DeepFaceLab as a digital human.