search

divamgupta / stable-diffusion-tensorflow

Stable Diffusion in TensorFlow / Keras
Other
1.58k stars 228 forks source link

Why prompt is limited to 77 #20

Open fengwang opened 2 years ago

fengwang commented 2 years ago

In a pipeline I replaced the pytorch version with this implementation, but found the maximum prompt is limited to 77. Is this a compromise for some reasons?

jorgemcgomes commented 2 years ago

The clip-vit-large-patch14 (https://huggingface.co/openai/clip-vit-large-patch14) model used by SD can only handle sequences of 77 tokens. It works like that in the original pytorch implementation as well. Anything longer than that gets silently truncated.

fengwang commented 2 years ago

The clip-vit-large-patch14 (https://huggingface.co/openai/clip-vit-large-patch14) model used by SD can only handle sequences of 77 tokens. It works like that in the original pytorch implementation as well. Anything longer than that gets silently truncated.

Thank you for the kind reply.

In this case, I would suggest truncating the the sequence to 77 tokens while giving a warning, instead of throw an assertion error from this place:

https://github.com/divamgupta/stable-diffusion-tensorflow/blob/cc32f5a4f958ff79d0b45b46b8f4ccc909e924d9/stable_diffusion_tf/stable_diffusion.py#L43

soten355 commented 1 year ago

The clip-vit-large-patch14 (https://huggingface.co/openai/clip-vit-large-patch14) model used by SD can only handle sequences of 77 tokens. It works like that in the original pytorch implementation as well. Anything longer than that gets silently truncated.

Thank you for the kind reply.

In this case, I would suggest truncating the the sequence to 77 tokens while giving a warning, instead of throw an assertion error from this place:

https://github.com/divamgupta/stable-diffusion-tensorflow/blob/cc32f5a4f958ff79d0b45b46b8f4ccc909e924d9/stable_diffusion_tf/stable_diffusion.py#L43

Any tips on how to truncate or where to get started?