Negative and weighted prompts

[bakkot]

First, this is really cool! I'm mostly getting complete gibberish prompts (e.g. aamaaamagranddaughter admire illustrations lmp profile halsey fortnite followart ️ ultimatefangraphics hounews) but they still reliably reproduce some sense of the original illustration.

Second: many tools now allow you to specify weights for parts of the prompts, including negative weights. Do you think it would be possible to generalize this technique to produce prompts with weights? These are still a little bit "hard", in the sense of being comprehensible for humans, but allow a much finer level of precision, particularly with negative weights.

[YuxinWenRick]

Hi, sorry for the late response. Thanks for your suggestion.

We found that the learned prompts can also be used for negative prompts. For example, in this notebook, we want to optimize the prompt of a grass image, which works pretty well, and then use this as a negative prompt. The generation with the learned negative prompt doesn't contain any greenery.

You could also try to flip the sign of the loss to generate the negative prompt on the target image, but when we tried this, we could notice any interesting improvement. We believe that in the high-dimension CLIP embedding space, there are so many solutions to the negative prompt of any given target image, so it's hard to find meaningful negative prompts.

Concerning the weights for specific parts, this could be interesting future work, but for now, it is not so clear what the optimal optimization target for this would be.