search

cloneofsimo / paint-with-words-sd

Implementation of Paint-with-words with Stable Diffusion : method from eDiff-I that let you generate image from text-labeled segmentation map.
MIT License
635 stars 52 forks source link

Commas, Periods and Tokens #41

Closed setothegreat closed 1 year ago

setothegreat commented 1 year ago

This isn't technically an issue, but I thought posting this here could help alleviate an issue some people might face.

Because of the way Paint with Words is implemented, you cannot use commas in your prompt for the reasons mentioned in the description of this GitHub. This can be an issue since commas are commonly used to separate tokens in a prompt. As such, a prompt without commas can have drastically less tokens, and thus the image generated can be both drastically less detailed, and drastically worse quality than an image with commas, as the model has less information to work with. In my testing without PWW enabled, the tokens used in the prompt I was testing went from 52 to 34 tokens after removing commas, and the output was far lower quality.

However, simply replacing your commas with periods seems to act perfectly fine as a substitute, and results in an equal number of tokens as when using commas. Further, whilst there is a tiny difference in very small details of the generation (clouds and other background details can be moved slightly, and other small, inconsequential changes) the image generated with the same seed is almost exactly the same, with the quality of the image being about equal as a result. Importantly, with this change you can then also implement the exact text with periods into the PWW obj field without it throwing up an error like it does if you use commas.

I thought I would bring this up since it isn't mentioned in the description of the GitHub, as the description simply says "don't use commas". I think modifying the description to instead say "replace commas with periods in both prompt and obj field" could be a much more helpful to people who don't know about this workaround.

lwchen6309 commented 1 year ago

Thanks for your great suggestion. I had replace "," by "@" in the latest version. Please let me know if there's other comments :)