pipeline_stable_diffusion.py trys to encode the prompt when there is no prompt and `prompt_embeds` and `negative_prompt_embeds` are given

[CaffreyR]

Describe the bug

pipeline_stable_diffusion.py trys to encode the prompt when there is no prompt and prompt_embeds and negative_prompt_embeds are given.

Reproduction

Use --pre_compute_text_embeddings when train a dreambooth scripts. Use #6135 by @sayakpaul updating the train_dreambooth_lora.py. Even if prompt_embeds and negative_prompt_embeds are proven to pipeline, it still wants to find prompt to encode and return Nonetype. So --pre_compute_text_embeddings can be used only in IF.

Logs

No response

System Info

• diffusers version: 0.24.0
• Platform: Linux-5.15.0-56-generic-x86_64-with-glibc2.31
• Python version: 3.9.18
• PyTorch version (GPU?): 1.11.0+cu115 (True)
• Huggingface_hub version: 0.19.4
• Transformers version: 4.36.1
• Accelerate version: 0.25.0
• xFormers version: not installed
• Using GPU in script?: Yes
• Using distributed or parallel set-up in script?: No

Who can help?

@yiyixuxu @DN6 @sayakpaul @patrickvonplaten

[CaffreyR]

Pull request regarding this #6202

[CaffreyR]

Hi @patrickvonplaten, if the modification of #6202 is wrong. Please see https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_lora.py#L1239 when prompt_embeds is given, the model does not give prompt to the pipeline. And it will return error when running it.

[CaffreyR]

Hi @sayakpaul , any ideas about it?

[DN6]

@CaffreyR Could you please provide a code example of the command you're running that produces the error.

[CaffreyR]

Hi @DN6 , please see https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth.py#L164 if args.pre_compute_text_embeddings , there is no "prompt" in the pipeline_args, so when pipeline_args to the pipeline, it report error.

[DN6]

@CaffreyR I meant could you please share the exact command you are running in your terminal that produces this error. It is hard to reproduce this with just a link to line in the training file.

[CaffreyR]

Hi @DN6 , please follow the instruction of https://github.com/huggingface/diffusers/tree/main/examples/dreambooth and use the code below to get dataset

from huggingface_hub import snapshot_download

local_dir = "./dog"
snapshot_download(
    "diffusers/dog-example",
    local_dir=local_dir, repo_type="dataset",
    ignore_patterns=".gitattributes",
)

and after that please use

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export INSTANCE_DIR="dog"
export OUTPUT_DIR="path-to-save-model"

accelerate launch train_dreambooth.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="a photo of sks dog" \
  --resolution=512 \
  --train_batch_size=1 \
  --gradient_accumulation_steps=1 \
  --learning_rate=5e-6 \
  --lr_scheduler="constant" \
  --lr_warmup_steps=0 \
  --max_train_steps=400 \
  --validation_prompt="a photo of sks dog on the beach" \
  --seed="0" \
  --push_to_hub \
  --pre_compute_text_embeddings \
  --num_validation_images=50

it uses --pre_compute_text_embeddings and error in validation function of "runwayml/stable-diffusion-v1-5" and when it comes to validation, it will report

Traceback (most recent call last):
  File "/data2/gw723/1/train_dreambooth.py", line 1429, in <module>
    main(args)
  File "/data2/gw723/1/train_dreambooth.py", line 1353, in main
    images = log_validation(
  File "/data2/gw723/1/train_dreambooth.py", line 176, in log_validation
    image = pipeline(**pipeline_args, num_inference_steps=25, generator=generator).images[0]
  File "/data2/gw723/anaconda3/envs/tmp/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/data2/gw723/1/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 933, in __call__
    prompt_embeds, negative_prompt_embeds = self.encode_prompt(
  File "/data2/gw723/1/diffusers/src/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 489, in encode_prompt
    unscale_lora_layers(self.text_encoder, lora_scale)
  File "/data2/gw723/1/diffusers/src/diffusers/utils/peft_utils.py", line 131, in unscale_lora_layers
    for module in model.modules():
AttributeError: 'NoneType' object has no attribute 'modules'

[CaffreyR]

Hi @DN6 , any ideas about it?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.