[Experiment] Retraining decoder to 256x256

[Clorr]

Hi guys,

I'm currently exploring some new ways of faceswapping. I wanted to share my last experiment on higher quality swaps which consists of modifying the decoder so that it outputs 256x256 images.

I assumed the following:

• input and output do not have to be the same size. Encoder does not need to have many details because anyhow it will erase a part of it
• The deeper autoencoders are, the harder they are to be trained.
• Training the encoder for ages won't give better quality result
• Decoder A is useless at some point
• Decoder is here to transform a low parameter input into a visible image, so it is the real part that is to be focused on for higher quality result.

So, I added some layers to the decoder and decided to train the decoder B specifically, by getting rid of decoder A and by setting encoder.trainable to False.

So far I'm getting this (loss of 0.029):

[babilio]

Interesting! I think your assumptions make sense and this looks very promising. How would I go about testing this. Can I use an old model and just train again with -t OriginalRetrain? Then when converting use -t OriginalRetrain or just -t Original?

Thanks!

[Clorr]

You can use an already trained model. The plugin will create a new decoder file that won't overwrite the existing one (=> the old decoder is not reloaded, at least for now, but the loss goes down much faster than for the initial training at least at the beginning).

For the converter, I did not manage yet to use the generated face at full resolution. I'll push an update when possible... (but yes, for the -t arg, you will have to use OriginalRetrain)

[Clorr]

Note that my goal was to have more details on output, but I don't think the simple decoder architecture will let me have what I want. However, we can still test this approach with much higher resolution (but then we will have to extract higher res faces)

[Clorr]

@babilio I pushed a convert, it is as experimental as the rest ;-)

[babilio]

I did a quick train and a convert and the convert turned even better than the training previews. It was a little blurry but I am working with a set of 250 images for B, so tentatively with bigger set or more training it would be better my loss was .02.

[iperov]

I also tried various decoders to reach more detailed face. No chance. I think its because NN learning to reach average of all warped crap we input to it.

look at gif showing warped input and output

even bent nose outputs straight nose.:

on the left looking eyes outputs straight looking:

because all averaged. So why we expecting sharp details in averaged faces ?

Latest Daddario ^ I trained with this decoder:

    def Decoder(self):
        input_ = Input(shape=(8, 8, 512))
        x = input_
        x = self.upscale(512)(x)
        x = self.upscale(256)(x)
        x = self.upscale(128)(x)
        x = self.upscale(64)(x)
        x = self.upscale(32)(x)
        x = Conv2D(3, kernel_size=5, padding='same', activation='sigmoid')(x)
        x = NearestNeighborDownsampler()(x)
        x = BicubicDownsampler()(x)        
        return KerasModel(input_, x)

[Link Removed]

[Clorr]

As I said in #249 , this is not the way we will have more crisp faces because at some point the loss function is the problem.

However I was thinking higher res images would help on some small size features like eyes that sometimes are not really well rendered (I'm not talking about the sight direction but more on the fact that in some places eyes are just a horrible mix of pixels, and with higher res, you get something more meaningful). I wanted to see the results, now I see it's limits, but I thought it could help some others.

Also the goal of the experiment was to try to freeze the encoder so we can try training deeper decoders and this part seems to work.

On the next step, I'm trying to document myself on how to use the residual loss to train some side network (like the mask in GAN), because this project is for me a way to learn deep learning more than doing actual faceswaps

[Clorr]

I updated the code with DSSIM loss function and res_block. Not very big improvments, but still better than previous. This is the result:

[kvrooman]

the issue of the "averaged face" pose/eye gaze direction/blur is due to the way the loss function is designed. differences in alignment ( double eyebrows, eye gaze, distorted or zoomed features, incorrect expression or pose) are lumped together with differences in texture ( color, lighting, gamma, white balance ) in the single loss function.

was reading a paper with some very interesting ideas on loss functions and feedback from seperate items... 1. landmark error, 2. texture error of the frontal flattened image, 3. percerputal error from the final warped image ( by similiarity functions or the return of a seperate CNN recognition model )

https://www.arxiv-vanity.com/papers/1701.04851/

[Clorr]

Thanks for the link, seems very interesting I'll try to read this ASAP. Note you can post this kind of info in faceswap-model repo, it is more dedicated to gather this kind of paper.

I totally agree that the loss is the problem now. The first layers are trained well, we just now have a problem on the very last layers that don't get the right info to adjust correctly. I'm currently trying simple solutions because I don't think we need very complicated solutions. In style transfer GAN, folks try to generate images for which they don't have a ground truth so creating a discriminator is a must. But here we have one, so the model should converge to this very ground truth...

[gessyoo]

Can somebody give me a quick explanation how to run the experimental retrain model or post the code? I tried inserting the experimental code in a cloned Original Model directory but got error messages.