[SD 1.5 LoRA Fine-Tuning] Is this the expected loss curve when fine-tuning SD with LoRA?

[codingperks]

Describe the bug

Hi there,

I'm currently fine-tuning SD1.5 on my machine using the diffusers LoRA script (detailed here) on the Pokemon BLIP dataset.

However, my (unsmoothed/smoothed) loss curve looks like this:

Are these the kinds of loss curves (which do not seem to decrease) par for the course with LoRA fine-tuning? What kind of loss curves should we expect with ideal conditions/parameters?

Thank you in advance for the help!

Reproduction

https://huggingface.co/docs/diffusers/training/lora

export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="/sddata/finetune/lora/pokemon"
export HUB_MODEL_ID="pokemon-lora"
export DATASET_NAME="lambdalabs/pokemon-blip-captions"

accelerate launch --mixed_precision="fp16"  train_text_to_image_lora.py \
  --pretrained_model_name_or_path=$MODEL_NAME \
  --dataset_name=$DATASET_NAME \
  --dataloader_num_workers=8 \
  --resolution=512 --center_crop --random_flip \
  --train_batch_size=1 \
  --gradient_accumulation_steps=4 \
  --max_train_steps=15000 \
  --learning_rate=1e-04 \
  --max_grad_norm=1 \
  --lr_scheduler="cosine" --lr_warmup_steps=0 \
  --output_dir=${OUTPUT_DIR} \
  --push_to_hub \
  --hub_model_id=${HUB_MODEL_ID} \
  --report_to=wandb \
  --checkpointing_steps=500 \
  --validation_prompt="A pokemon with blue eyes." \
  --seed=1337

Logs

No response

System Info

Nvidia RTX 3080

• diffusers version: 0.18.0.dev0
• Platform: Linux-5.19.0-46-generic-x86_64-with-glibc2.35
• Python version: 3.10.11
• PyTorch version (GPU?): 2.0.0 (True)
• Huggingface_hub version: 0.15.1
• Transformers version: 4.30.2
• Accelerate version: 0.20.3
• xFormers version: 0.0.20

Who can help?

@williamberman @patrickvonplaten @saya

[BitPhinix]

Yeah, it’s expected. If you look at the average loss curve you should see a small decrease over time, but only a very small one

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.