search

ThereforeGames / txt2mask

Automatically create masks for Stable Diffusion inpainting using natural language.
519 stars 40 forks source link

I have a request but...It can be challenging! #16

Open FlowDownTheRiver opened 2 years ago

FlowDownTheRiver commented 2 years ago

I don't know if this is possible or not but for example I want to change the cloth of multiple people across multiple images,or I want to change all the faces to certain someone's face with txt2image masking.Like deepfakes so to say. I may have sequence of images or random images doesn't matter.

Screenshot 2022-10-03 at 09-23-20 Stable Diffusion

So what I have in mind is as in the picture,I will mask out the faces,I will write a prompt and do a batch inpainting by utilizing the automatic masking of txt2mask.

I am not sure if you can do it alone or maybe automatic himself needs to be involved,but this is a great idea imo and maybe a new harder task or challenge for you :)

Please tell me it can be done.

In short what I am asking is a batch inpainting feature powered by clip masking with txt2mask.

FlowDownTheRiver commented 2 years ago

Screenshot 2022-10-03 at 09-40-42 Stable Diffusion It shall support all the features of inpainting tab including inpaint at full resolution.

FlowDownTheRiver commented 2 years ago

More ideas...

Wildcard Support :

Maybe also wildcards support with it. Given in here https://github.com/jtkelm2/stable-diffusion-webui-1/blob/master/scripts/wildcards.py and this also support that wildcard script https://github.com/Lopyter/stable-soup-prompts.

So the idea is for example if there is an exhibit,an art gallery with pictures, we automatically mask out "wall hanging pictures" and we replace them randomly from a txt file which is the wildcard here.We can add anything to that txt file which is very flexible and create new txt files as wildcards or change the faces of people with random celebrities by auto masking and using our celebrity wildcard list.

Sorry for making the request with divided parts but the ideas doesn't come all at once. It can be much more advanced this way.

Loopback with wildcard support:

A bit more complex can be multi pass with wildcards,like change the "man in the picture" to become a celebrity from wildcard list , and loop back that picture as a source and this time use the second mask "the car" to become a ferrari or something else from our car list of wildcards. And this doesn't have to be done with wildcards but a loopback function can do some multiple masks in order or random order.

I have some crazy ideas but don't have the skillset do make them :) But I hope you can,if you have time.

jujpenabe commented 2 years ago

One thing I really hope can be implemented is the batch inpainting, the "inpainting at full resolution" is really important imo.