Open tin2tin opened 2 months ago
Hi, max_sequence_length
it's only for the T5 Text Encoder, the warning you're getting refers to the Clip Text Encoders which will still have the 77 token limit.
This is in the documentation where it says:
The prompt with the CLIP Text Encoders is still truncated to the 77 token limit.
If you want to use long prompts and even weighting, you can use sd-embed which is a very good and small library.
So, doing this is a no go?
image = pipe(
prompt="",
prompt_3=prompt,
negative_prompt="",
num_inference_steps=28,
guidance_scale=4.5,
max_sequence_length=512,
).images[0]
image.save('sd3-single-file-t5-fp8.png')
That works, the warning and what gets truncated is the prompt for the Clip Text Encoders, those are the same ones as the SDXL models, the T5 is the big model that gives you the prompt adherence.
So in short:
prompt
, prompt_2
gets truncated at 77 tokens
prompt_3
uses the max_sequence_length
You can use a separate prompt for each if you want, like in this example.
In my tests this is what works best. I didn't see any improvements with sending also a big prompt to the Clip Text Encoders.
Also to clear the misunderstanding, your first example also works, even if the prompt gets truncated you will still see the image to get the additional details of the long prompt because the T5 prompt doesn't get truncated.
Oh, so the full prompt will be used even if there is the truncated message? (If so, that IS confusing)
I think what also confused me is that the PixArt Sigma models comes with a much higher tokens limit in the prompt option, without any further ado.
I know it seems complicated, we're thinking of doing a glossary so people can understand this better.
Pirxart uses a single Text Encoder, so it doesn't have this problem, it uses only the T5 which is a LLM, so it only handles text.
The Clip Text Encoders are smaller models which in simpler terms mix the images with text, they don't have the capacity of an LLM.
SD3 uses 3 text encoders, the T5 which allows long prompts and 2 Clip Text Encoders which have a low token limit (same as SD 1.5 and SDXL).
What is happening is that the prompt you're sending is sent to the three of them at the same time and gets truncated just for the Clip Text Encoders at the 77 limit, the T5 still receives the full prompt, same as pixart so it still generates the image with the full promp.
We didn't hide the warning because in reality the prompt is getting truncated for those models so it's better if people are aware of this instead of hiding it and make users think the full prompt is also being used by those models.
Describe the bug
max_sequence_length=512 doesn't have any effect on SD3 # of tokens
Reproduction
Logs
System Info
Diffusers 0.29.2 & 0.30.0 dev Win 11
Who can help?
@yiyixuxu @sayakpaul @DN6 @asomoza