search

deepfakes / faceswap-playground

User dedicated repo for the faceswap project
305 stars 194 forks source link

Has anyone had good results with masked GAN? #98

Closed Jack29913 closed 5 years ago

Jack29913 commented 6 years ago

I'm training a GAN-64 model expecting to get face obstructions handled but no luck. Options, perception loss and instance normalization are on and it has been trained for 20k ite with large image set (A-16k, B-9k). Do you think is it because i included those frames (I thought model needed to learn from those frames) or bad port or maybe it just works for simple cases and greatly exaggerated in original gan repo? I'm really curious that if there is anyone who succeed in this.

bryanlyon commented 6 years ago

My GAN experience is limited, but my suggestion on general ai tasks is to do a long training on related data but not the actual data (so faces of the same person, but from different videos) you're going to use, then after a good training, do another, shorter training on the actual video data you're swapping. This gives the model a limited opportunity to overtrain on the dates you're using.

You may be giving it too many faces, the more you give it, the longer it'll take to train, even if it's the same person. It's better to use a few hundred good shots instead. Make sure to get a variety of lighting and directions for the great results.

On Mar 13, 2018 3:05 PM, "Apollo122" notifications@github.com wrote:

I'm training a GAN-64 model expecting to get face obstructions handled but no luck. Options, perception loss and instance normalization are on and it has been trained for 20k ite with large image set (A-16k, B-9k). Do you think is it because i included those frames (I thought model needed to learn from those frames) or bad port or maybe it just works for simple cases and greatly exaggerated in original gan repo? I'm really curious that if there is anyone who succeed in this.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/deepfakes/faceswap-playground/issues/98, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEuwdj6UWy5-SiWm57voVDKWOKJchWDks5teEKVgaJpZM4Sphug .

ruah1984 commented 6 years ago

i have train up to 50k ite before for a few days, preview looks not bad, but after convert with mask , looks bad and not actually like what shaoanlu show in his preview, maybe we miss up something

Jack29913 commented 6 years ago

Trying these new trainers has been completely waste of time. First I tried IAE results were horrible, and now GAN. In my opinion original trainer is still best and stable. And I think GAN merge was a mistake. Couple of people including me reported there are problems with GAN regarding mask. If mask has problems there is a good possibility that other parts have problems too considering this is a port. It got merged anyway and it's useless

bryanlyon commented 6 years ago

All of our work is experimental, don't use anything here for another that matters. This is not a magical tool, it's a learning experience.

That said, GAN and IAE are both new adds. We'll be fixing bugging at least for a few weeks before they'll be stable.

If you NEED results and want them consistent, then yes, stick with original. The other models are not as stable as the original model yet.

As we move on, the models and software will get better, that's the point. We'd never get anywhere if we didn't try new models.

On Mar 14, 2018 00:22, "Apollo122" notifications@github.com wrote:

Trying these new trainers has been completely waste of time. First I tried IAE results were horrible, and now GAN. In my opinion original trainer is still best and stable. And I think GAN merge was a mistake. Couple of people including me reportted there are problems with GAN regarding mask. If mask has problems there is a good possibility that other parts have problems too considering this is a port. It got merged anyway and it's useless

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/deepfakes/faceswap-playground/issues/98#issuecomment-372927500, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEuwWj0Ao1d9m7BjyHrnS2Arad5fjWaks5teMVMgaJpZM4Sphug .

Jack29913 commented 6 years ago

If they are not stable why did they get merged to master then? I agree they are experimental, I tested the GAN PR. I'm just saying it is a mistake to merge trainers before knowing that for sure they are working and stable.

ruah1984 commented 6 years ago

I try shaoanlu GAN and compare with this GAN Repo, with 15k samples,>50k itr unfortunately cant get a better face. i think if GAN willl need to train more and more. i will advice don't look at the preview window, preview window looks good, convert not means good. we shall try some sample convert after run certain times

bryanlyon commented 6 years ago

They got merged so everyone could use them. They're optional and we're doing our best to support them. We'll get them there, just give us time.

On Mar 14, 2018 01:07, "ruah1984" notifications@github.com wrote:

I try shaoanlu GAN and compare with this GAN Repo, with 15k samples,>50k itr unfortunately cant get a better face. i think if GAN willl need to train more and more. i will advice don't look at the preview window, preview window looks good, convert not means good. we shall try some sample convert after run certain times

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/deepfakes/faceswap-playground/issues/98#issuecomment-372936418, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEuwSZ5-v9LbdfvWOjZLvRcEBv1a3w1ks5teM-ogaJpZM4Sphug .

bryanlyon commented 6 years ago

We should be able to match the convert to what you see in preview, just a lot to figure out and check. The preview is a representation of what the model sees, we're just having trouble getting it out, it's a technical problem, and a surmountable one.

On Mar 14, 2018 01:07, "ruah1984" notifications@github.com wrote:

I try shaoanlu GAN and compare with this GAN Repo, with 15k samples,>50k itr unfortunately cant get a better face. i think if GAN willl need to train more and more. i will advice don't look at the preview window, preview window looks good, convert not means good. we shall try some sample convert after run certain times

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/deepfakes/faceswap-playground/issues/98#issuecomment-372936418, or mute the thread https://github.com/notifications/unsubscribe-auth/ADEuwSZ5-v9LbdfvWOjZLvRcEBv1a3w1ks5teM-ogaJpZM4Sphug .

ruah1984 commented 6 years ago

I know is on the way, i not too worry about it. For me is too bad that, https://github.com/deepfakes/faceswap/pull/246 option not include to the latest repo. i have try this before, it actually help a little bit on the training , with same itr .@iperov do a great job, hope he can re-open back this and merge with the master.

Clorr commented 6 years ago

@Apollo122 I understand your point as I think you see the project as an out-of-the-box ready tool for end-users. I do not see it this way.

First, the work of contributors, whether they add code or just help others (as you do) is much more important to me than leechers who come complain and don't even bother to let an improvment after they have succeeded. So someone who did something, pushes a PR and says "I have put some of my brain time on this, now I think it is ready, and I can't go much further" is already to be considered. If it's work is not perfect, I don't care as long as it is good enough.

Second thing, by providing a pluggable architecture, I sent a message to programmers to say "Hey guys, come in! You can also add something of your own, and if not, you can also get inspiration from previous users and improve things". This way I try to lower the number of hard forks like GAN or DFaker (which started before this repo has grown) but also I offer a possibility to new programmers to be able to see the different approaches and maybe get inspired to see a better solution. Also, what's not perfect today can help someone and can be improved tomorrow; what is not published won't ever be improved. And if you set too high expectations in accepting new code, you will discourage many people from publishing.

Also I think if Original was perfect we wouldn't have to discuss about this. Maybe it is the best for most cases today, but still it needs refining. And that's why I want people to publish (and this is also why I put [DRAFT] and [experiment] PRs)

Maybe some day we will have tens of active contributors, many projects based on the repo, and this day the politics will change. However for now, it is still a WIP

Jack29913 commented 6 years ago

@Clorr I understand what you are saying and yeah people who help others or push code are valuable and I would never want to discouraged them in fact I argued some people who do that in repo. But I don't agree that merging to master would make the code better. Merging to master should be last step though I'm seeing this approach with new PR's and it's a good thing. I actually specifically criticize the GAN merge even @oatssss commented that he was getting different results from the original repo. In my opinion It needed more time and wasn't ready thats all what I'm saying. Maybe we can create another branch and merge these experimental stuff there? Most open-source projects have stable and experimental branches.

Clorr commented 6 years ago

@oatss had much work on his PR and I didn't want to let him on it forever. It's a very big work to do that and we don't have so much skilled contributors. So it is fair enough that his work is merged and that he can move on to something else (or can improve GAN with peace of mind).

I agree that at some point we can have an experimental branch but I let that to contributors who want to work this way. And the problem with experimental branch + few users is that anyway, you will have people waiting for the code to be in master and that will only test and feedback when code is in master.

For plugins, as it is optional and that users can deliberately choose their prefered way of training it, one should try a plugin, see if he is satisfied with it. If not, he can contribute or move on to another option.

iperov commented 6 years ago

@shaoanlu is this good mask ?

python_2018-03-17_16-36-26

Jack29913 commented 6 years ago

@iperov have you tried with obstructions such as hands in front of the face? Do they get blurred or you can see the hands clearly?

iperov commented 6 years ago

not yet

iperov commented 6 years ago

@oatssss is there any color bug with converting ?

python_2018-03-17_18-40-11

converted: 2018-03-17_18-46-36

match hist off

iperov commented 6 years ago

I trained only 3 hours :)

iperov commented 6 years ago

lol GAN128 even without mask gives superior result as on 3rd column 'masked' window. even better than dfaker. Face properly scaled. but convert uses 3rd column of 'raw' window python_2018-03-17_23-58-55

Face properly scaled vs faceswap original:

chrome_2018-03-18_00-06-05

iperov commented 6 years ago

no mask :(

python_2018-03-18_09-45-29

2018-03-18_09-44-57

so original model still best general purpose face swapper

iperov commented 6 years ago

conclusion: GAN still unusable for me. Nothing changed since initial GAN commit without 128 and vggface.

Jack29913 commented 6 years ago

I'm training GAN128 model. I trained for 16k without PL and mask refinement. I'll continue with with PL for 10k then train more with refinement. But I don't think it will work. Will share the final results here

iperov commented 6 years ago

better mask with improved traindata generator? https://github.com/deepfakes/faceswap/pull/307 python_2018-03-23_00-32-21

Jack29913 commented 6 years ago

Looks like an improvement but I don't know. You should try it with occlusions such as hands and see if it's handled. @oatssss commented that he was trying to get better results by altering the generator so it may be the way

oatssss commented 6 years ago

@iperov, shaoanlu made a mistake in his preview code: "raw" and "masked" are actually inverted (raw preview shows masked result, masked preview shows raw result). As for the color bug, try turning on seamless cloning, It should color correct so it's not as patchy as in the preview.

Are you guys able to get masks that properly handle occlusion with shaoanlu's? Something to try is change shaoanlu's notebook code so the model save/load code is compatible with this repo (it should be like a 3-line change), then use the converter here with the model trained via shaoanlu's.

Sorry guys, I haven't really been working on the project recently

Jack29913 commented 6 years ago

It worked with mask refinement!!! Its not included in master so I did a PR. Check the hand on raw preview. And yes its a bug like @oatssss mentioned. I was looking at the masked all the time :) Although training took ages.

Maybe with @iperov 's new data generator it takes less.

@Clorr please merge the PR I did.

[Image removed]

Clorr commented 6 years ago

done ;-)

Jack29913 commented 6 years ago

Yes they are needed. I shared the options I used for the training. I also disabled and enabled mix_up for a couple of iterations but I don't think it mattered. What I can say about GAN is that you must have patience and faith :)

Note that I didn't try these options for GAN-64. Also there's no mask refinement option for it in the original repo. Maybe it works without it but I don't know.

Jack29913 commented 6 years ago

Btw it seems that shaoanlu did some updates for v2 (one we are using). Automated training for perceptual loss and switching to proper loss functions at predefined iterations. @oatssss fyi

oatssss commented 6 years ago

@Apollo122 how was your conversion with those results you got? Were they at least on par with the original model? Thanks for the heads up on shaoanlu's stuff I'll look into it. Also, anyone can feel free to do the gan stuff.

Jack29913 commented 6 years ago

@oatssss unfortunately it wasn't good enough, original is still better. The masks I got like dotted pictures, i mean it's not smooth you can check the image above, maybe that interfered. Btw someone did a scripted version for gan. You may check PR's in gan repo.

ruah1984 commented 6 years ago

@Apollo122 base on the result you share, may i know how is the result after convert??

Jack29913 commented 6 years ago

@ruah1984 it wasn't as good as original. Mask wasn't smooth enough and because it is applied on face it directly effects the quality