support a phrase/seed per batch element

[matpalm]

current the seed used seeds the entire batch latent, it's more flexible to instead use a seed per batch element. for the same reason each batch element should allow a different prompt. this allows better iteration on prompt/seed as independent pieces.

( this change also include the latent seeding bug proposed in a prior, now closed, PR )

e.g. sample some seeds for a fixed prompt

prompts = ["a blue cabin in the forest next to a lake."] * 10
seeds = list(range(len(prompts)))

imgs = generator.generate(
    prompts,
    seeds,
...

pick seed=8 as a favourite and iterate on prompt

prompts = []
for colour in ['blue', 'dark blue', 'light blue', 'green and blue', 'green']:
    prompts.append(f"a {colour} cabin in the forest next to a lake.")
seeds = [5] * len(prompts)

imgs = generator.generate(
    prompts,
    seeds,
...

[fnurl]

Why not go all the way and change the parameters num_steps, unconditional_guidance_scale and temperature to lists as well so those parameters can be varied across the batch?

[fnurl]

I tried using your branch with 13 prompts and the last 5 images came out as noise:

    # Prompt and seed copied from
    # https://lexica.art/?prompt=715596cf-84bd-497f-8413-6e9bb8f39c5e
    main_prompt = "cat seahorse fursona, autistic bisexual graphic designer, attractive fluffy humanoid character design, sharp focus, weirdcore voidpunk digital art by artgerm, akihiko yoshida, louis wain, simon stalenhag, wlop, noah bradley, furaffinity, artstation hd, trending on deviantart"
    seed = 4030098432

    # create prompts where each prompt adds one of the comma separated components
    prompts = []
    for p_comp in main_prompt.split(','):
        if not prompts:
            prompts.append(p_comp.strip())
        else:
            prompts.append(f"{prompts[-1]}, {p_comp.strip()}")
    seeds = [seed for _ in range(len(prompts))]

Here is the noise image (all 5 were the same)

Running .generate() once for each prompt (single prompt in list) however was fine.

[matpalm]

Why not go all the way and change the parameters num_steps, unconditional_guidance_scale and temperature to lists as well so those parameters can be varied across the batch?

temperature is actually never used; see https://github.com/divamgupta/stable-diffusion-tensorflow/issues/21

unconditional_guidance_scale could be added

num_steps is the number of diffusion steps, not a parameter that's part of the forward pass. how would you vary this across the batch? bucketing (by distinct values) or something?

[matpalm]

I tried using your branch with 13 prompts and the last 5 images came out as noise:

this works for me? weird. the noise ones are what you get with a low num steps. not sure what's happending for you..

[jacobkreider]

temperature is actually never used; see #21

Sorry, what do you mean here? I run a loop that changes temp value (among other things) and see differences in style...are you saying that if I kept temp constant and ran the same number of iterations, I would expect the same variation?

[matpalm]

yeah, the sampling step loop isn't fully ported.

https://github.com/divamgupta/stable-diffusion-tensorflow/blob/2873079b49eda63f41267c69b825e03cbcb2c308/stable_diffusion_tf/stable_diffusion.py#L117-L126

temperature is only ever used to calculate noise, and noise isn't added to x_prev, and even if it was added, it would only be 0 anyway, since sigma_t is always 0.

[innat]

@matpalm stable diffusion is added to keras-cv. If you think this PR would be an improvement, would you please condersing to keras-cv? https://github.com/keras-team/keras-cv

[matpalm]

yeah you've copied across the bugs too, eg see the sigma all zero => no noise case

On Mon, 26 Sept 2022, 01:51 Mohammed Innat, @.***> wrote:

@matpalm https://github.com/matpalm stable diffusion is added to keras-cv. If you think this PR would be an improvement, would you please condersing to keras-cv? https://github.com/keras-team/keras-cv

— Reply to this email directly, view it on GitHub https://github.com/divamgupta/stable-diffusion-tensorflow/pull/32#issuecomment-1257222145, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAFILOK33I4LNZ7E5OV6LDWABYIZANCNFSM6AAAAAAQTOODZQ . You are receiving this because you were mentioned.Message ID: @.***>