tg-bomze
commented
10 months ago Hey, @damian0815. Thanks for the Сompel. It's a very useful library.
I noticed that now long prompts are handled differently than in AUTOMATIC111111's webUI. Like you said, they break tokens into chunks, disregarding the separating punctuation (",", ".", ":"). I've written a function that should bring the generation results closer to those obtained in the webUI from AUTOMATIC1111.
For starters:
import torch
from compel import Compel
from diffusers import StableDiffusionPipeline, DDIMScheduler
pipeline = StableDiffusionPipeline.from_single_file(
"https://huggingface.co/runwayml/stable-diffusion-v1-5/blob/main/v1-5-pruned-emaonly.safetensors",
torch_dtype=torch.float16,
)
pipeline.scheduler = DDIMScheduler.from_config(pipeline.scheduler.config)
pipeline.to("cuda")
compel = Compel(tokenizer=pipeline.tokenizer, text_encoder=pipeline.text_encoder)
prompt = "A highly detailed matte painting of retro futurist mount fuji hosting the olympics, aerial photography, ultrawide lens, by dan mumford, yusuke murata, makoto shinkai, ross tran, cosmic, heavenly, god rays, intricate detail, cinematic, 8 k, cel shaded, unreal engine, featured on artstation, pixiv, hd sharp 3d model vray render in pixar squareenix game anime manga toriyama miyazaki style trending on pixiv skeb"
prompt_emb = compel.build_conditioning_tensor(prompt)
negative_prompt = "deformed iris, deformed pupils, text, worst quality, low quality, jpeg artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, camera"
negative_prompt_emb = compel.build_conditioning_tensor(negative_prompt)
image = pipeline(
prompt_embeds=prompt_emb,
negative_prompt_embeds=neg_prompt_embeds,
width=512,
height=512,
num_inference_steps=30,
num_images_per_prompt=1,
guidance_scale=7,
generator=torch.Generator(device='cuda').manual_seed(42)
).images[0]This is the result without truncate:
And here's the result with truncate:
In the webUI from AUTOMATIC111111 if we use the same parameters and the same model, we get:
Here is the necessary function that encodes the prompt and negative prompt correctly:
from torch.nn.utils.rnn import pad_sequence
import torch.nn.functional as F
eos_token = pipeline.tokenizer.eos_token_id
bos_token = pipeline.tokenizer.bos_token_id
comma_token = pipeline.tokenizer.encode(',')[1]
dot_token = pipeline.tokenizer.encode('.')[1]
colon_token = pipeline.tokenizer.encode(':')[1]
max_tokens = pipeline.tokenizer.model_max_length - 2
def concat_tensor(t):
t_list = torch.split(t, 1, dim=0)
t = torch.cat(t_list, dim=1)
return t
def tokenize_line(line, pipeline): # split into chunks
actual_tokens = pipeline.tokenizer.encode(line.strip(), truncation=False)[1:-1]
chunks = []
chunk = []
for item in actual_tokens:
chunk.append(item)
if len(chunk) == max_tokens:
if chunk[-1] not in [comma_token, dot_token, colon_token]:
for i in range(max_tokens-1, -1, -1):
if chunk[i] in [comma_token, dot_token, colon_token]:
temp_chank = [bos_token] + chunk[:i+1] + [eos_token] * (max_tokens+1 - len(chunk[:i+1]))
chunks.append(temp_chank)
chunk = chunk[i+1:]
break
else:
temp_chank = [bos_token] + chunk + [eos_token] * (max_tokens+1 - len(chunk))
chunks.append(temp_chank)
chunk = []
else:
temp_chank = [bos_token] + chunk + [eos_token] * (max_tokens+1 - len(chunk))
chunks.append(temp_chank)
chunk = []
if chunk:
temp_chank = [bos_token] + chunk + [eos_token] * (max_tokens+1 - len(chunk))
chunks.append(temp_chank)
return chunks
def get_embeddings_for_long_prompt(prompt, negative_prompt, pipeline):
prompt_tokens = tokenize_line(prompt, pipeline)
neg_prompt_tokens = tokenize_line(negative_prompt, pipeline)
if len(prompt_tokens) != len(neg_prompt_tokens):
diff = abs(len(prompt_tokens) - len(neg_prompt_tokens))
if len(prompt_tokens) > len(neg_prompt_tokens):
neg_prompt_tokens.extend([[bos_token] + [eos_token]*(max_tokens+1)]*diff)
elif len(neg_prompt_tokens) > len(prompt_tokens):
prompt_tokens.extend([[bos_token] + [eos_token]*(max_tokens+1)]*diff)
prompt_embeds = []
neg_prompt_embeds = []
for chunks_batch in [prompt_tokens, neg_prompt_tokens]:
tokens_tensor = pad_sequence([torch.tensor(chunk) for chunk in chunks_batch], batch_first=True).to(pipeline.device)
attention_mask = torch.ones_like(tokens_tensor).to(pipeline.device) # Create attention mask
model_output = pipeline.text_encoder(tokens_tensor, attention_mask=attention_mask) # Removed weights argument
if chunks_batch is prompt_tokens:
prompt_embeds.append(model_output.last_hidden_state)
else:
neg_prompt_embeds.append(model_output.last_hidden_state)
prompt_embeds = torch.cat(prompt_embeds, dim=1)
neg_prompt_embeds = torch.cat(neg_prompt_embeds, dim=1)
return concat_tensor(prompt_embeds), concat_tensor(neg_prompt_embeds)Next, call the function:
prompt_emb, neg_prompt_embeds = get_embeddings_for_long_prompt(prompt, negative_prompt, pipeline)And this is the result:
Here's a link to Colab to test it out. The code probably looks overloaded, but I hope it's useful to you. Compel is just an awesome tool and it lacks proper handling of long prompts.
damian0815
sarmientoj24
segalinc
andreineamtu
at the moment compel hard-splits prompts at 75 tokens. because self-attention does not work outside of a single 75 token chunk, this can cause degraded generation especially if a word is split in the middle.
compel should backtrack from 75 tokens to find the previous
.;or,character (searching in that order) and split there.