Diffusion update breaks train_dreambooth scripts

[EandrewJones]

Describe the bug

Last night I was running the dreambooth fine-tuning scripts on colab fine but today after Diffusion release 0.9.0 landed, the training script breaks.

Nothing in my code has changed except now when I launch the training script on accelerate I see the error below.

Running %pip install -qq git+https://github.com/ShivamShrirao/diffusers seems to install version Diffusion 0.9.0 and all I can think is that this has introduced a breaking change in the training script.

Is there anyway I can pin diffusers version to 0.8.1 for now when installing this github?

Reproduction

Try running the fine-tuning example notebook.

Logs

Traceback (most recent call last):
  File "train_dreambooth.py", line 822, in <module>
    main(args)
  File "train_dreambooth.py", line 458, in main
    revision=args.revision
  File "/usr/local/lib/python3.7/dist-packages/diffusers/pipeline_utils.py", line 680, in from_pretrained
    model = pipeline_class(**init_kwargs)
  File "/usr/local/lib/python3.7/dist-packages/diffusers/pipelines/stable_diffusion/pipeline_stable_diffusion.py", line 143, in __init__
    feature_extractor=feature_extractor,
  File "/usr/local/lib/python3.7/dist-packages/diffusers/pipeline_utils.py", line 149, in register_modules
    library = module.__module__.split(".")[0]
AttributeError: 'list' object has no attribute '__module__'
Traceback (most recent call last):
  File "/usr/local/bin/accelerate", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/accelerate_cli.py", line 43, in main
    args.func(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 837, in launch_command
    simple_launcher(args)
  File "/usr/local/lib/python3.7/dist-packages/accelerate/commands/launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['/usr/bin/python3', 'train_dreambooth.py', '--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--pretrained_vae_name_or_path=stabilityai/sd-vae-ft-mse', '--output_dir=/content/drive/MyDrive/stable_diffusion_weights/@potion', '--revision=fp16', '--with_prior_preservation', '--prior_loss_weight=1.0', '--seed=1337', '--resolution=512', '--train_batch_size=1', '--train_text_encoder', '--mixed_precision=fp16', '--use_8bit_adam', '--gradient_accumulation_steps=1', '--learning_rate=2e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=200', '--sample_batch_size=4', '--max_train_steps=2000', '--save_interval=250', '--save_guidance_scale=20', '--save_infer_steps=250', '--save_sample_prompt=fire @potion with dragon inside bottle, perfectly ornated, intricate details, 3d render vray, uhd, beautiful, trending', '--concepts_list=concepts_list.json', '--not_cache_latents']' returned non-zero exit status 1.

### System Info

diffusers==0.9.0
OS: Google Colab Pro (Linux)
GPU: Tesla T4

After installing the github repo, `!pip show diffusers` returns:

Name: diffusers Version: 0.9.0.dev0 Summary: Diffusers Home-page: https://github.com/huggingface/diffusers Author: The HuggingFace team Author-email: patrick@huggingface.co License: Apache Location: /usr/local/lib/python3.7/dist-packages Requires: filelock, regex, Pillow, requests, importlib-metadata, huggingface-hub, numpy Required-by:

[rmac85]

Same issue here using: https://colab.research.google.com/github/ShivamShrirao/diffusers/blob/main/examples/dreambooth/DreamBooth_Stable_Diffusion.ipynb?authuser=0&pli=1#scrollTo=gW15FjffdTID

All I could do was import my model to a different colab and run it there. I have not tried to train a new one yet but I assume would cause the same issue generating samples after training. Did you find a fix yet?

[EandrewJones]

@rmac85 No because I was in the middle of fine-tuning lots of different models :/

Since it's a fork, it's tracking all the updates directly from diffusers branch. We might need to pin the pip install directly toa commit prior to v0.9.0 for now. Will update if I find a commit hash that works.

[MysticSpiderman]

Also ran into this issue where the DreamBooth Stable Diffusion Collab no longer works. Any ideas of this is something that can be fixed? Following this for updates.

In the meantime trying out the fast-dreamboth collab.

Thanks for all your hard work.

[ShivamShrirao]

Hey, Sorry I didn't catch that during the update. Will fix it in a bit.

[rmac85]

Thanks, pease update here when you do.

[MysticSpiderman]

Beautiful Thanks ShivamShriaro. Hoping things run smooth. My fiance and I are working on a project and have been in a good creative flow thanks to the collab. Thanks again.

[naoyaikeda]

I specified

!wget -q https://github.com/ShivamShrirao/diffusers/raw/main/examples/dreambooth/train_dreambooth.py !wget -q https://github.com/ShivamShrirao/diffusers/raw/main/scripts/convert_diffusers_to_original_stable_diffusion.py %pip install -qq git+https://github.com/ShivamShrirao/diffusers@7684518377180879a6e45611820e92c9dec7d4b6 %pip install -q -U --pre triton %pip install -q accelerate==0.12.0 transformers ftfy bitsandbytes gradio natsort

This is maybe Okay.

[rmac85]

I specified

!wget -q https://github.com/ShivamShrirao/diffusers/raw/main/examples/dreambooth/train_dreambooth.py !wget -q https://github.com/ShivamShrirao/diffusers/raw/main/scripts/convert_diffusers_to_original_stable_diffusion.py %pip install -qq git+https://github.com/ShivamShrirao/diffusers@7684518377180879a6e45611820e92c9dec7d4b6 %pip install -q -U --pre triton %pip install -q accelerate==0.12.0 transformers ftfy bitsandbytes gradio natsort

This is maybe Okay.

Where to put that and does it need to replace anything?

[EandrewJones]

@rmac85 try pinning the diffusers install to the commit just prior to v0.9.0dev merge into main:

%pip install -qq git+https://github.com/ShivamShrirao/diffusers@cecdd8bdd1c0ac902483e464f40ebdaa91f3fe13

This is working for me.

You would change the line in the training notebook under install requirements: https://github.com/ShivamShrirao/diffusers/blob/7dc594706b15f51bcc2a69bf6a5ad17e39959774/examples/dreambooth/DreamBooth_Stable_Diffusion.ipynb

As per Naoyaikeda's comment , it looks like any commit prior to the update should work.

[MysticSpiderman]

Quick side question - What Sample Method does this collab use?

[rmac85]

Okay, thanks so much. I'm sure Shivam will get it fixed for us soon as he said either way. I am an AI addict though so not my patience. LOL Will give it a try.

[ShivamShrirao]

Should be fixed in 4affee020e85eecff56e002be77270daf4e5cba7.

[ShivamShrirao]

@MysticSpiderman DDIM by default but u can use others like Euler, euler a, DPM by importing them from diffusers library and replacing the scheduler in pipeline.

[rmac85]

Shivam, I am kind of a dummy, do you own the colab notebook or is it just somebody else's based on your code? I'm just wondering if you can provide detailed instruction on how to import the other schedulers in the colab, like exactly what line to put and where. Sorry to be a pest, you must be very busy. Thanks so much for your great work.

[rmac85]

In inference I tried: from diffusers import StableDiffusionPipeline, EuleraScheduler - doesn't work.

I'm not sure the right way to put it.

[rmac85]

Okay I got it myself, it's called EulerAncestralDiscreteScheduler Had to remove clip_sample and set_alpha_to_one