[Question] Can face swap models run in browser?

[seranus]

Was wondering if models could run in current frameworks like KerasJS or WebDNN, if can I'd appreciate some advice on setup.

I'm a developer but my machine learning knowledge is at a basic level.

Got some free time and resources I was thinking of making a web site where people could share their models and run them on simple gifs directly in browser without the need to install anything. In the end they could publish the result gif.

It could be interesting for developers as a playground to test or showcase and general public to try out face swaping and see what there getting into before setting up the python code.

If someone wants to join up i wont say no ether.

[oatssss]

Seems very doable! We could have a backend that converts uploaded models for KerasJS then send those to the client (not sure how large these model files will be though). Maybe throw in an upvote system as well to cull out the bad ones over time.

One thing to note though, is that the models require only face shots as input. So users would need to perform extraction/merging themselves (maybe there are already JS frameworks for this purpose that we can take advantage of).

I really like this idea actually, what tech stack do you have in mind for the site? I have a bit of experience in web development with meteor and react.

[seranus]

I was thinking of starting with something simple like main page with published gifs, view filters and upvotes, user login with upload history and create gif page where you can upload or reuse models/gifs. We also would need at least 3 workers, one for model conversion, one for gif import encoding and one for gif publishing. It's something that I could prototype in a relatively short time.

Stuff that I don't know how to do is converting the model and making it run in browser.

For the faces we could use a face detector like hog and input it into the model. After that make a preview with stitched face over the video to avoid unnecessary load on the backend. User could the click publish and send it to a worker that would merge it and save it online.

First we should make a prototype with just creating on a single model in browser just to see how the whole thing flows and whats possible and whats not.

My background is mostly c# I was thinking of doing the fronted in angular and backend c#/python and hosting it on Azure..

[oatssss]

KerasJS provides a python script that can be run by a worker to convert models to KerasJS format.

What would the gif import encoding worker do?

[seranus]

That's makes it easier, can i convert faceswap encoder and decoder with KerasJS encoder you linked?

Will something like this work? Do i need to specify the quantize param?

encoder.py encoder.h5 -n encoder

For gif importing I was thinking of serializing gifs to the same resolution with the same frame rate if the if high frame/resolution gifs end up being considerably slower.

[oatssss]

I haven't tried, but that looks correct, the optional arguments may need to come before the model file, dunno. The docs indicate that quantization compresses models to 1/4 the size so I'd recommend using it!

[seranus]

To get the model for KerasJS i did self.autoencoder_A.save() in https://github.com/deepfakes/faceswap/blob/master/plugins/Model_Original.py#L23

Managed to generate the bin model with the encoder you linked.

After trying to import it to KerasJS I'm getting error Layer Model specified in model configuration is not implemented!

I'm using the sample model Trump-Cage that you can download with the main repo, not sure if its the face swap model issue or is it KerasJS issue. Stuck atm

Error is defined here: https://github.com/transcranial/keras-js/blob/master/src/Model.js#L319

[oatssss]

You shouldn't need to add a save() call in Model_Original.py. If you run the trainer (in a command prompt) with:

python "faceswap.py" train ^
  -A "path\to\face\a\aligned" ^
  -B "path\to\face\b\aligned" ^
  -m "path\to\model\dir" ^
  -t Original ^
  -bs 32

it'll generate models for you in path\to\model\dir.

I'll need to look into this error more. At first glance it seems like KerasJS doesn't support the architecture of the model, but according to the error the layer's name is "Model" which seems funky to me.

[seranus]

I thought that regular training would just saves weights and that isn't enough information. I'm getting an error when using encoder.h5 or decoder.h5. When using model save KerasJS encoder worked without errors.

File "h5py\h5a.pyx", line 77, in h5py.h5a.open KeyError: "Can't open attribute (can't locate attribute: 'model_config')"

Model I'm using https://www.dropbox.com/s/jwh0zjqvpe2eooq/autoencoder.bin?dl=0

[oatssss]

Oh actually yea you're right, we'll need to save the autoencoders and not just encoder/decoder_A/decoder_B cuz we can't reconstruct the autoencoder once it's in js I believe. Not sure why the KerasJS converter doesn't work for for the individual pieces though since they're still models.

[seranus]

You miss understood my previous post i ran encoder and decoder with save_weights.

Today I retrained the decoders and encoder and saved the models. When loading into KerasJS I get an error:

[Model] Layer PixelShuffler specified in model configuration is not implemented!

How big of an issue if that, going to try it on webDNN

[oatssss]

PixelShuffler isn't an implented layer in KerasJS because it's a custom layer. I dunno if KerasJS supports custom layers, but there's nothing in the docs at the moment :(

With WebDNN, I don't think the model will work out-of-the-box. but it looks like there's example code we can follow to get the layer working... it doesn't seem trivial though.

[seranus]

After investigating webDnn it seems to be lacking the documentation and some Keras layers like permute are not imaplanted

It seems that currently the best bet to get this working is by hacking the PixelShuffler layer into KerasJs cos it seems to be a lot simpler to add a layer. Made an empty layer and imported the model successfully so for the loading it seems getting that layer to work is only issue https://github.com/seranus/keras-js/blob/master/src/layers/custom/PixelShuffler.js

Currently trying to get some test input data from face swap so I can debug try/error do the layer. Taking the data from face variable

from https://github.com/deepfakes/faceswap/blob/master/plugins/Convert_Masked.py#L66

[seranus]

I managed to write PixelShuffler layer for cpu, currently only working with images and sample trump cage mode. It should work on any face swap model except GAN since it uses more custom layers

Here's the model runner https://seranus.github.io/faceswap-browser/faceswap-browser/dist/index.html

[Clorr]

Hi @seranus

You can integrate this in the main repo if you want or at least open an issue there so that more people see your work

[seranus]

@Clorr Thanks for the response, sure I'll post in the main repo. Just give me a few days I'd like to make a KerasJS model conversion script and add browse local files functionality so people could start using it. If it's fine with the community I'd like to keep it in my repo and making it a helper tool for making face swap video