[SDXL] Sample generation during training not working

[kenjiqq]

Here are examples of samples that was generated when i trained a short test dreambooth for just 400 steps(sample every 100 steps)

But when i ran the the minimal sdxl inference script on the model after 400 steps i got

I think part of the problem is samples are generated at a fixed 512x512, sdxl did not generate that good images for 512x512 in general. But don't think that is the main problem as i tried just changing that in the sampling code and images are still messed up

[markrmiller]

Interesting. On my attempt I got the same type of samples but also go then with the minimal infer.

[wzgrx]

SDXL have been made public officially as well now, it is there on stabilityai profile of hugging face: https://huggingface.co/stabilityai/stable-diffusion-xl-base-0.9 https://huggingface.co/stabilityai/stable-diffusion-xl-refiner-0.9

[markrmiller]

Maybe the issue I had was due to using Adam 8-bit. Missed the note it didn't work.

So I changed to recommend. Model wouldn't train. Found the notebook for this and aligned settings. Model wouldn't train. Problem was gradient checkpointing and inputs having no grad. I get OOM without check pointing, so tried various things to enable, couldn't. Eventually I just hacked torch’s checkpoint python file to default the reentrant arg to false (I believe) which they claim will change to that soon anyway in the comments, and then model trained.

The training samples look somewhat reasonable now, though the qaulity and content seem a little off still.

[OGSally]

These recent changes might have fixed this issue

https://github.com/kohya-ss/sd-scripts/commit/77ec70d145deb30cba0a9d972d9aee762fbb7268 https://github.com/kohya-ss/sd-scripts/commit/c2ceb6de5fc861513582642edf87834a01ce84a2

[Enzzer]

+1 on this, can't get correct samples generated during SDXL LoRa training.

[Gitterman69]

samples every n step doesnt work unfortunately (latest version). samples get generated after each epoch though.... training lora on sdxl 1.0 base model

[jmattthew]

The default sample size is 512x512, and in SDXL, which makes samples similar to OP's images. Using the parameters --w 1024, --h 1024 in prompts.txt creates sample images that resemble the prompt more but still aren't correct. Also the paramter --n (negative prompt) isn't working: it's being incorporated into the prompt.

[JiePKU]

I got the same error. The generated images (even before training, using --sample_at_first) is not good, like the illustrated images. Help!

[nelapetrzelkova]

Check the content of prompt.txt. Instead of the actual prompt there is a path to the file itself. Workaround for now is to manually rewrite the content with your prompt after the prompt.txt file is created.