search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.53k stars 5.29k forks source link

Train sd3 controlnet but result is bad #9674

Closed xduzhangjiayu closed 1 hour ago

xduzhangjiayu commented 2 hours ago

Describe the bug

Hi, I tried to train sd3 controlnet with examples\controlnet\train_controlnet_sd3.py, but it seems the validation results are really bad, and the training loss was oscillating all the time, it really confused me for a long time, could you please give some advice? Thanks a lot !

Here are some validation results and train loss: validation output on steps 2900: validate loss function in tensorboard loss

Reproduction

accelerate launch examples/controlnet/train_controlnet_sd3.py --pretrained_model_name_or_path=stable-diffusion-3-medium-diffusers --output_dir=output --resolution=1024 --learning_rate=1e-5 --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" --validation_prompt "red circle with blue background" "cyan circle with brown floral background" --train_batch_size=1 --gradient_checkpointing --gradient_accumulation_steps=4

Logs

No response

System Info

diffusers: 0.31.0.dev (latest) torch: 2.4.0

Who can help?

@sayakpaul @DN6 @asomoza

sayakpaul commented 1 hour ago

This should belong to discussions. Ccing @PromeAIpro who contributed the script.