search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.72k stars 5.31k forks source link

[Advanced Dreambooth] Textual Inversion token is weakened/ignored by negative prompt #6703

Open levi opened 9 months ago

levi commented 9 months ago

Describe the bug

A LoRA + TI token pair trained with the advanced dreambooth script doesn't work with negative prompts.

In my testing, the validations all work perfectly if a negative prompt is not used. When it is included, the subjected trained with the token is often entirely ignored or looks nothing like the original subject.

Reproduction

  1. Train a LoRA + TI with the advanced dreambooth script using an instance prompt like "A photo of TOK".
  2. Perform validation prompt like "A photo of TOK" to see valid training results
  3. Perform same validation, but with an additional negative prompt like "distorted, disfigured, blurry"
  4. Notice the TOK subject is often ignored and when not ignored, retains less characteristics of the original subject

Logs

Continuing my investigation here: https://github.com/huggingface/diffusers/issues/6685#issuecomment-1908679012

System Info

Main branch of diffusers

Who can help?

@linoytsaban

yiyixuxu commented 8 months ago

cc @linoytsaban @sayakpaul here

github-actions[bot] commented 7 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

sayakpaul commented 7 months ago

@linoytsaban in case you missed it.

github-actions[bot] commented 6 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions[bot] commented 5 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.