search

FeliMe / AudioStyleNet

This repository contains the code for my master thesis on Emotion-Aware Facial Animation
146 stars 29 forks source link

Running demo with custom image #4

Open Sarmyt opened 3 years ago

Sarmyt commented 3 years ago

Interested in running the colab demo with a custom image, would I need to use resnetEncoder to convert my image into latent space. If so, where can I find the pretrained weights for the resnetEncoder?

kevinhuangxf commented 3 years ago

Could you share the resetnetEncoder pretrained model? I tried to use other stylegan-encoder to get my custom image. But its lipsync result is not as ideal as the demo latent vectors. I guess it may need the resetnetEncoder. Thanks for your fancy work.

FeliMe commented 3 years ago

There is no one universal resnetEncoder that works for every image. Using projector.py for single images gives the best results and is the way to go.

bduvenhage commented 3 years ago

Hi @FeliMe

I've used projector.py to find latent face vectors for my own images and the resulting re-renders look great.

However, the resulting latent vectors don't always animate well. In many cases the lip motion is suppressed and sometimes there is 'crosstalk' into other parts of the face or shoulders.

I noticed that your training data seem to be from a 'Aligned256' folder. Do you perhaps have some examples of how you crop and align faces in preparation for the projection step so that it matches the data expected by your your model?

Warm regards, Bernardt

bduvenhage commented 3 years ago

Hi @FeliMe

I used your align_face.py to automatically crop out a 256x256 face, but still some of the projected latent vectors are not as animatable as others. I'll keep digging.

Regards, Bernardt