basujindal / stable-diffusion

Optimized Stable Diffusion modified to run on lower GPU VRAM
Other
3.14k stars 467 forks source link

Are negative weights supported? #243

Open rivered opened 1 year ago

rivered commented 1 year ago

I am currently using txt2img.py, and it is giving the weirdest results with many deformations of limbs and all the known problems. I have tried to add to the prompt the following to get rid of deformations, but the brackets are completely ignored, and actually the words within are used as prompt and deformations get even worse.

python optimizedSD/optimized_txt2img.py --prompt "Woman on the beach posing in sunlight [bad anatomy extra legs extra arms extra fingers poorly drawn hands poorly drawn feet disfigured out of frame tiling bad art deformed mutated]" --H 1028 --W 1028 --n_iter 5 --n_samples 1 --ddim_steps 8

To solve this problem, it is highly advised to add a negative prompt as outlined here. However, when I review the txt2img.py code, it is difficult to understand what to change...

All what is needed is the following: `c = model.get_learned_conditioning(prompts) uc = model.get_learned_conditioning(negative_prompts)

samplesddim, = sampler.sample(conditioning=c, unconditional_conditioning=uc, [...])`

However when I review the optimized_txt2img.py the following code is found: `all_samples = list() for n in trange(opt.n_iter, desc="Sampling"): for prompts in tqdm(data, desc="data"):

        sample_path = os.path.join(outpath, "_".join(re.split(":| ", prompts[0])))[:150]
        os.makedirs(sample_path, exist_ok=True)
        base_count = len(os.listdir(sample_path))

        with precision_scope("cuda"):
            modelCS.to(opt.device)
            uc = None
            if opt.scale != 1.0:
                uc = modelCS.get_learned_conditioning(batch_size * [""])
            if isinstance(prompts, tuple):
                prompts = list(prompts)

            subprompts, weights = split_weighted_subprompts(prompts[0])
            if len(subprompts) > 1:
                c = torch.zeros_like(uc)
                totalWeight = sum(weights)
                # normalize each "sub prompt" and add it
                for i in range(len(subprompts)):
                    weight = weights[i]
                    # if not skip_normalize:
                    weight = weight / totalWeight
                    c = torch.add(c, modelCS.get_learned_conditioning(subprompts[i]), alpha=weight)
            else:
                c = modelCS.get_learned_conditioning(prompts)`

Can this code be modified, to take negative prompts? Is there anyone using the optimized_txt2img.py succesfully without adding negative prompts?