New various results.

[iperov]

(sry bad english)

I keep working on new OpenDeepFaceSwap every day. I will share it of course.

I choosed worst case data where source face is brighter, dest face is darker, skin tone differents. Source face has bright blue background which is BAD for original faceswap and fakeapp.

All models targets same epoch (~4-5h training) with same batch size.

First look at work of 'super improved DF full face match warper':

^it warps src face to nearest dst face and randomizes it. 'Super improved' because fixed all bugs which exist in original DF warper and warper changed from buggy scipy.interpolator to scipy Delaunay triangulator-warper. Dst face not warps to src, because OpenDeepFaceSwap will swap only src->dst, so why overfill NN with useless data.

Original 128 with super improved DF full face match warper:

Original 128 with DSSIM loss func (DF model) with super improved DF full face match warper:

^ better than original.

DSSIM loss makes result better than without it, also it suppress of learning background, so in theory more neurons will learn face instead of background.

Also we can see original model just averages source faces, therefore source face has half closed eyes.

IAE 128 with DSSIM loss func with super improved DF full face match warper:

^ bettter than DF, face more properly aligned, eyes opened and looking straight, while all facial features saved.

IAE 128 experimental with DSSIM loss func with super improved DF full face match warper:

Compare in detail IAE128 and IAE128 experimental:

In IAE128 experimental we can see better supressing of blue background of source and better skin fade matching.

In IAE128 experimental I found new way how to make IAE better. Main idea is gather NOT only 4x4 features in intermediate layers, but also 8x8.

def Intermediate(self, is_low_mem):
    input_layer = self.keras.layers.Input(shape=(None, 4 * 4 * 1024))

    neck = self.keras.layers.Dense(1024)(input_layer)

    x4x4 = self.keras.layers.Dense(4 * 4 * 512)(neck)
    x4x4 = self.keras.layers.Reshape((4, 4, 512))(x4x4)

    x8x8 = self.keras.layers.Dense(8 * 8 * 256)(neck)
    x8x8 = self.keras.layers.Reshape((8, 8, 256))(x8x8)

and then add it in decoder:

x = upscale(self.keras, x4x4, 512)
x = res(self.keras, x, 512)

x = self.keras.layers.Add()([x, x8x8]) 

but it consumes more VRAM. I would like to add 16x16 features but I cant due to lack of vram of my card with 6GB.

IAE training history https://webm.video/i/mX5TCJ.mp4

[iperov]

@dfaker, in last mp4 video can you say something about flickering holes in masks ?

[iperov]

IAE128 experimental 2:

better than IAE128 experimental 1 !!

[dfaker]

@iperov It seems to be identical in both axes for both the mask and the rgb. That'd lead me to guess that it's due to the network learning some bias in the averaged transformations of all of the samples in the last trained batch.

[edit] oops sorry you mean the mask, I thought you meant the remaining jitters even at the end of training.

[iperov]

@dfaker any suggestions how to fix it ?

[dfaker]

@iperov That's the sort of thing that lead me to put the mask and rgb layers into separate decoders, I saw both this (an 'edge detection' on edge like features in the face causing voids in the mask) and the mask layer bleeding thorough to to add a color cast or lighting effect to the rgb face data.

Sadly my best solution was, as you saw, to split them into two parallel decoders, a simpler one would probably suffice for the mask however.

[iperov]

@dfaker um... Actually this mask flickering bug in your model. I use same separated mask trainer.

def Decoder(self, is_low_mem):

        input  = self.keras.layers.Input(shape=(4, 4, 1024))
        x = input
        x = upscale(self.keras, x, 512)
        x = res(self.keras, x, 512)        

        x = upscale(self.keras, x, 256)
        x = res(self.keras, x, 256)        

        x = upscale(self.keras, x, 128)
        x = res(self.keras, x, 128)        

        x = upscale(self.keras, x, 64)
        x = res(self.keras, x, 64)        

        x = upscale(self.keras, x, 32)
        x = res(self.keras, x, 32)

        x = self.keras.layers.convolutional.Conv2D(3, kernel_size=5, padding='same', activation='sigmoid')(x)

        y = input  #mask decoder
        y = upscale(self.keras, y, 256)
        y = upscale(self.keras, y, 128)
        y = upscale(self.keras, y, 64)
        y = upscale(self.keras, y, 32)
        y = upscale(self.keras, y, 16)
        y = self.keras.layers.convolutional.Conv2D( 1, kernel_size=5, padding='same', activation='sigmoid' )(y)

        return self.keras.models.Model( [input], [x,y])

[iperov]

looks like it depends on dimensions of upscale layers.

Finally IAE experimental 3 ! Best model of all time.

this video demonstrates result: F128 -> DF -> IAE original -> IAE 1 -> IAE 2 -> IAE 3

https://webm.video/i/n4ddXk.mp4

[iperov]

Total size of .h5 files of IAE 3 is 792MB.

[iperov]

@acsaga thanks for your original IAE.

[dfaker]

@iperov I was under the impression that you had made some innovations in the network itself, not just re-using the existing model. Perhaps your tweaks are all in the encoder not just the sample generation.

Are you suggesting that the areas of dark within the mask of your failed cases are some kind of deconvolution artifact? They don't at all have the classic look one would expect.

[iperov]

Perhaps your tweaks are all in the encoder not just the sample generation.

yeah 1) your match warper 2) smart sample feeder by sorted yaw to prevent model overfit with only one yaw. 3) your dssim loss, because original IAE without dssim 4) in IAE3:

    def Encoder(self, input_layer, is_low_mem):
        x64 = input_layer
        x32 = conv(self.keras, x64, 128)
        x16 = conv(self.keras, x32, 256)
        x8 = conv(self.keras, x16, 512)
        x4 = conv(self.keras, x8, 1024)

        return self.keras.models.Model(input_layer, 
                [  
                    self.keras.layers.Flatten()(x32),
                    self.keras.layers.Flatten()(x16),
                    self.keras.layers.Flatten()(x8),
                    self.keras.layers.Flatten()(x4)
                ]
            )

    def Intermediate(self, is_low_mem):
        input_x32 = self.keras.layers.Input(shape=(None, 32 * 32 * 128))
        input_x16 = self.keras.layers.Input(shape=(None, 16 * 16 * 256))
        input_x8 = self.keras.layers.Input(shape=(None, 8 * 8 * 512))
        input_x4 = self.keras.layers.Input(shape=(None, 4 * 4 * 1024))

        x32 = self.keras.layers.Dense(64)(input_x32)
        x16 = self.keras.layers.Dense(128)(input_x16)
        x8 = self.keras.layers.Dense(256)(input_x8)
        x4 = self.keras.layers.Dense(512)(input_x4)

        x = self.keras.layers.Concatenate()( [x32, x16, x8, x4] )

        x = self.keras.layers.Dense(4 * 4 * 512)(x)
        x = self.keras.layers.Reshape((4, 4, 512))(x)

        return self.keras.models.Model( [input_x32, input_x16,  input_x8, input_x4], [x]  )

and it can be upgraded when new cards with more VRAM will appear in future. This models works on 6GB with only 2 batch size lool.

Are you suggesting that the areas of dark within the mask of your failed cases are some kind of deconvolution artifact?

problem almost fixed when I increased dims to 512 from 256.

        y = upscale(self.keras, y, 512)
        y = upscale(self.keras, y, 256)
        y = upscale(self.keras, y, 128)
        y = upscale(self.keras, y, 64)
        y = upscale(self.keras, y, 32)

now much better.

https://webm.video/i/Xb3qDs.mp4