Can not reproduce the output

[htrvu]

Hello there. I followed your provided code and data to train the concept at data/dog. However, I can not achieve as good output as showed in your paper. Am I missed something?

My outputs:

• Learing rate 1e-4, batch size 1, spl weight is 1.0

• Learing rate 2e-5, batch size 1, spl weight is 1.0(equivalent to learning rate 1e-5 and batch size 2 in your setting)

Training scripts:

export CLS_TOKEN="dog"
export MODEL_NAME="runwayml/stable-diffusion-v1-5"
export OUTPUT_DIR="./ckpt/${CLS_TOKEN}_cls_1e-4"
export INSTANCE_DIR="./data/${CLS_TOKEN}"

accelerate launch train_class_diffusion.py \
  --pretrained_model_name_or_path=$MODEL_NAME  \
  --instance_data_dir=$INSTANCE_DIR \
  --output_dir=$OUTPUT_DIR \
  --instance_prompt="a photo of a <new1> ${CLS_TOKEN}"  \
  --resolution=512  \
  --train_batch_size=1  \
  --learning_rate=1e-4  \
  --lr_warmup_steps=0 \
  --max_train_steps=500 \
  --scale_lr \
  --hflip  \
  --modifier_token "<new1>" \
  --no_safe_serialization \
  --use_spl \
  --spl_weight=1 \
  --cls_token "${CLS_TOKEN}" \
  --validation_steps=50 \
  --validation_prompt="a <new1> ${CLS_TOKEN} is swimming" \
  --tracker_name "custom-diffusion-multi"

[Rbrq03]

Hello @htrvu, Thanks for interest.

• For the first experimental settings, I think it won't produce a good result as you use a 10 times learning rate and tunning-based subject-driven generation are sensitive for the hyper-parameters. For more details you can see https://huggingface.co/docs/diffusers/training/dreambooth#script-parameters https://huggingface.co/blog/dreambooth They shows that tunning-based subject-driven generation(like dreambooth) will produce results of very different quality with even slight hyper-parameters change.
• For the second experimental settings, we mentions that all experiment are conducted on 2 * Nvidia RTX4090 in our paper. So the learning rate should be 4e-5 in my environment by the argument --scale_lr. I think you can try to use this learning rate first considering the result you show seems don't fit to the custom dataset, though I am not sure if 4e-5 will be a proper learning rate for 1 gpu and batch 1.

The following are the results I produce in your caption.

If you continue to experience issues, feel free to reach out.

[htrvu]

Thanks for your quick reply. The results are better now!

[Rbrq03]

Glad to hear that!