search

neuralchen / SimSwap

An arbitrary face-swapping framework on images and videos with one single trained model!
Other
4.54k stars 892 forks source link

❓ Can we change Hair Color and such? - (not issue) #59

Closed AlonDan closed 3 years ago

AlonDan commented 3 years ago

I've just update to the newest version. While I follow the Preparation I've notice something ✨NEW✨ that now SimSwap uses face parsing so I've downloaded the model from the link and put it inside the correct directory as explained.

I've tried the --use_mask for no bounding box and it's BEAUTIFUL! so far so good.

Since SimSwap uses THIS now... Does it mean we can actually go farther and tweak some other masks such as Hair color? (maybe other parts as well?)

Face-Parsing.PyTorch

If so... what are the commands that I will need to add, that will be nice thing to test as long as it's not complicated since I'm not a programmer.

❓- Can we change SPECIFIC MASK PARTS colors? ❓- Can we decide which SPECIFIC MASK PARTS will be included or not? (selected parts of the face)

Original Color 1 Color 2

instant-high commented 3 years ago

No. I don't think it is possible without major changes. I managed to select what not to swap. Eg. Eyes...

NNNNAI commented 3 years ago

You can change the hair but I think changing the hair color may lead to the swapped face looks like more unreal.

instant-high commented 3 years ago

I've tried the above makeup repo a few month ago but not sure how to do in simswap.?

NNNNAI commented 3 years ago

@instant-high May be you can use the hair mask to do some magic.

instant-high commented 3 years ago

No. Shoulf be impossible since you can't "swap" the hair. The trained model don't recognizes the hair at all? Maybe it would mean to start a totally new SimSwap based on zllrunning face parsing ??

Btw. can the face outline be changed during swapping? What I want to say is that if the face in the video is small and the one in the nisource picture more round shaped this is not transferred... so sometimes the result is not very recoginzeable

NNNNAI commented 3 years ago

Unfortunately, face outline is unable to change in current simswap pipeline.

aesanchezgh commented 3 years ago

No. Shoulf be impossible since you can't "swap" the hair. The trained model don't recognizes the hair at all? Maybe it would mean to start a totally new SimSwap based on zllrunning face parsing ??

Btw. can the face outline be changed during swapping? What I want to say is that if the face in the video is small and the one in the nisource picture more round shaped this is not transferred... so sometimes the result is not very recoginzeable

Why not use a preprocess step and change hair color before the face swap?

instant-high commented 3 years ago

Could also be done before saving the final video. Problem is face parsing module (makeup) needs a square image or video frame containing the head. But the hair is partly outside the "square face bounding box" processed by simswap. So you have to detect and align it again .... don't know.

instant-high commented 3 years ago

I've made some temporal code changes to display the soft mask generated by face parsing. Hair is not inside the square images

instant-high commented 3 years ago

As an experiment I roughly inserted the zllrunning makeup.pytorch (https://github.com/zllrunning/face-makeup.PyTorch) to simswap. It's very slow because I think it's running on CPU (Dont't know so far how to change that, I'm more familiar to VisualBasic...) Here's a screenshot of an arbitrary example. Swap and Makeup. It also could perform makeup only.....

Parts and colors can be changed following the instructions from here : https://github.com/zllrunning/face-parsing.PyTorch

frame_0000054

Edit: seems already running on GPU