adieyal / sd-dynamic-prompts

A custom script for AUTOMATIC1111/stable-diffusion-webui to implement a tiny template language for random prompt generation
MIT License
2.02k stars 261 forks source link

Using a certain wildcard seems to inject parameters into image generation that ruin images - particularly with Pony v6. #823

Open a-l-e-x-d-s-9 opened 1 week ago

a-l-e-x-d-s-9 commented 1 week ago

It's very strange problem, but I was able to pinpoint and reduce it to a single wildcard with 2 lines. This problem discoverable mainly with Pony v6 checkpoint - probably due to Pony v6 being particularly sensitive to the change. Please use __prompttest_\ wildcard - it has 2 prompts: prompt_test.txt

Use the following parameters for the image and batch size 3:

__prompt_test__
Negative prompt: open mouth, bad quality, poor quality, disfigured, jpg, toy, bad anatomy, missing limbs, missing fingers, ugly, scary, watermark
Steps: 20, Sampler: DPM++ 2M, Schedule type: Karras, CFG scale: 7, Seed: 10, Size: 640x960, Model hash: 67ab2fd8ec, Model: ponyDiffusionV6XL_v6StartWithThisOne, RNG: CPU, Template: __prompt_test__, Negative Template: "open mouth, bad quality, poor quality, disfigured, jpg, toy, bad anatomy, missing limbs, missing fingers, ugly, scary, watermark", Emphasis: No norm, Version: v1.10.1

And you should get image that look like this: 20240922_234221_m 67ab2fd8ec_se 10_st 20_c 7_640x960 Notice the artifacts and very distorted image that you get. If you replace the prompt with the prompt it supposedly using: illustration, digital art, wild life, nature, trees, grass, bushes, flowers, mountain, wild nature, digital painting, medium shot, sharp focus, focused, cgsociety, analytical art, detailed, popart style, (cherrmous style:0.8), You will get this image instead: 20240922_234510_m 67ab2fd8ec_se 10_st 20_c 7_640x960 We can see a proper flowers, and whole image is lacking artifacts. The image that uses wildcard has artifacts that indicate something very wrong with image parameters. I think it's related to: Auto 1111 Attention/Emphasis Auto 1111 Alternating Words Which are used in my wildcards prompts. I have a lot of complex prompts in wildcard file that using emphasis, alternating words and other structures supported by Auto 1111. I suspect that Dynamic Prompts confusing the syntax with something else and injecting parameters into generated images - causing artifacts in the results. It's not something that noticeable with most checkpoints, but Pony v6 seems to be particularly vulnerable to these undesired changes. The problem exist with Auto 1111 and Forge. And it's not a new issue. This issue not influence 100% of the seeds, and some seeds seem to work ok. With the provided parameters, those are a few seeds that showing the issue: 10,14,15,18,19.

a-l-e-x-d-s-9 commented 1 week ago

I tried to simplify the prompts in the wildcard - doesn't seem to help. But using batch size 1 seems to make a fine image each time.

a-l-e-x-d-s-9 commented 3 days ago

@teamolhuang I appreciate the attempt to help. Yes, the file has two lines - two prompts, I wrote it in the bug report. The exact seeds that have the issue specified, particularly seed 10 - which clearly has prompt from line 1. Both of the prompts from file are working perfectly fine when written directly in the prompt. The issue appears only when dynamic prompts extension is used. I explained exactly how to reproduce it, and I already reproduced it on another system with another user.