huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.95k stars 5.35k forks source link

pipeline_stable_diffusion.py trys to encode the prompt when there is no prompt and `prompt_embeds` and `negative_prompt_embeds` are given #6201

Open CaffreyR opened 10 months ago

CaffreyR commented 10 months ago

Describe the bug

pipeline_stable_diffusion.py trys to encode the prompt when there is no prompt and prompt_embeds and negative_prompt_embeds are given.

Reproduction

Use --pre_compute_text_embeddings when train a dreambooth scripts. Use #6135 by @sayakpaul updating the train_dreambooth_lora.py. Even if prompt_embeds and negative_prompt_embeds are proven to pipeline, it still wants to find prompt to encode and return Nonetype. So --pre_compute_text_embeddings can be used only in IF.

Logs

No response

System Info

Who can help?

@yiyixuxu @DN6 @sayakpaul @patrickvonplaten

CaffreyR commented 10 months ago

Pull request regarding this #6202

CaffreyR commented 10 months ago

Hi @patrickvonplaten, if the modification of #6202 is wrong. Please see https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py#L1239 when prompt_embeds is given, the model does not give prompt to the pipeline. And it will return error when running it.

CaffreyR commented 10 months ago

Hi @sayakpaul , any ideas about it?

DN6 commented 10 months ago

@CaffreyR Could you please provide a code example of the command you're running that produces the error.

CaffreyR commented 9 months ago

Hi @DN6 , please see https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py#L164 if args.pre_compute_text_embeddings , there is no "prompt" in the pipeline_args, so when pipeline_args to the pipeline, it report error.

DN6 commented 9 months ago

@CaffreyR I meant could you please share the exact command you are running in your terminal that produces this error. It is hard to reproduce this with just a link to line in the training file.

CaffreyR commented 9 months ago

Hi @DN6 , please follow the instruction of https://github.com/huggingface/diffusers/tree/main/examples/dreambooth and use the code below to get dataset

from huggingface_hub import snapshot_download

local_dir = "./dog"
snapshot_download(
    "diffusers/dog-example",
    local_dir=local_dir, repo_type="dataset",
    ignore_patterns=".gitattributes",
)

and after that please use

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export INSTANCE_DIR="dog"
export OUTPUT_DIR="path-to-save-model"

accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="a photo of sks dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=400 \
  --validation_prompt="a photo of sks dog on the beach" \
  --seed="0" \
  --push_to_hub \
  --pre_compute_text_embeddings \
  --num_validation_images=50

it uses --pre_compute_text_embeddings and error in validation function of "runwayml/stable-diffusion-v1-5" and when it comes to validation, it will report

Traceback (most recent call last):
  File "/data2/gw723/1/train_dreambooth.py", line 1429, in <module>
    main(args)
  File "/data2/gw723/1/train_dreambooth.py", line 1353, in main
    images = log_validation(
  File "/data2/gw723/1/train_dreambooth.py", line 176, in log_validation
    image = pipeline(**pipeline_args, num_inference_steps=25, generator=generator).images[0]
  File "/data2/gw723/anaconda3/envs/tmp/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data2/gw723/1/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 933, in __call__
    prompt_embeds, negative_prompt_embeds = self.encode_prompt(
  File "/data2/gw723/1/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 489, in encode_prompt
    unscale_lora_layers(self.text_encoder, lora_scale)
  File "/data2/gw723/1/diffusers/src/diffusers/utils/peft_utils.py", line 131, in unscale_lora_layers
    for module in model.modules():
AttributeError: 'NoneType' object has no attribute 'modules'
CaffreyR commented 9 months ago

Hi @DN6 , any ideas about it?

github-actions[bot] commented 8 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.