TypeError thrown during save_weights() due to safety_checker = (None, None)

To fix the below console output and error, I have added if (save_method_name is not None): in the save_pretrained method before line 194 in pipeline_utils.py to cause it to skip trying to save safety_checker when it is (None, None).

Modified pipeline_utils.py:

177        for pipeline_component_name in model_index_dict.keys():
178            sub_model = getattr(self, pipeline_component_name)
179            model_cls = sub_model.__class__
180
181            save_method_name = None
182            # search for the model's base class in LOADABLE_CLASSES
183            for library_name, library_classes in LOADABLE_CLASSES.items():
184                library = importlib.import_module(library_name)
185                for base_class, save_load_methods in library_classes.items():
186                    class_candidate = getattr(library, base_class)
187                    if issubclass(model_cls, class_candidate):
188                        # if we found a suitable base class in LOADABLE_CLASSES then grab its save method
189                        save_method_name = save_load_methods[0]
190                        break
191                if save_method_name is not None:
192                    break
193
194            if (save_method_name is not None):
195                save_method = getattr(sub_model, save_method_name)
196                save_method(os.path.join(save_directory, pipeline_component_name))

Console output before modification:

(base) D:\Downloads\diffusers\examples\dreambooth>accelerate launch train_dreambooth.py   --pretrained_model_name_or_path="CompVis/stable-diffusion-v1-4"    --instance_data_dir="./training"   --class_data_dir="./classes"   --output_dir="./output"   --instance_prompt="a photo of xyz dog"   --class_prompt="a photo of a dog"   --resolution=512   --train_batch_size=1   --gradient_accumulation_steps=1   --learning_rate=5e-6   --lr_scheduler="constant"   --lr_warmup_steps=0   --num_class_images=200   --max_train_steps=20
The following values were not passed to `accelerate launch` and had defaults used instead:
        `--num_cpu_threads_per_process` was set to `14` to improve out-of-box performance
To avoid this warning pass in values for each of the problematic parameters or run `accelerate config`.
The config attributes {'set_alpha_to_one': False, 'skip_prk_steps': True, 'steps_offset': 1} were passed to DDPMScheduler, but are not expected and will be ignored. Please verify your scheduler_config.json configuration file.
Caching latents: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:16<00:00,  2.31s/it]
Fetching 16 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 2667.18it/s]
You have passed `None` for safety_checker to disable its functionality in <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'>. Note that this might lead to problems when using <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> and is not recommended.
You have disabled the safety checker for <class 'diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.StableDiffusionPipeline'> by passing `safety_checker=None`. Ensure that you abide to the conditions of the Stable Diffusion license and do not expose unfiltered results in services or applications open to the public. Both the diffusers team and Hugging Face strongly recommend to keep the safety filter enabled in all public facing circumstances, disabling it only for use-cases that involve analyzing network behavior or auditing its results. For more information, please have a look at https://github.com/huggingface/diffusers/pull/254 .
pipeline.save_pretrained(./output\20)
Traceback (most recent call last):
  File "train_dreambooth.py", line 812, in <module>
    main(args)
  File "train_dreambooth.py", line 805, in main
    save_weights(global_step)
  File "train_dreambooth.py", line 691, in save_weights
    pipeline.save_pretrained(save_dir)
  File "e:\anaconda3\lib\site-packages\diffusers\pipeline_utils.py", line 194, in save_pretrained
    save_method = getattr(sub_model, save_method_name)
TypeError: getattr(): attribute name must be string
Steps: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20/20 [08:29<00:00, 25.47s/it, loss=0.193, lr=5e-6]
Traceback (most recent call last):
  File "e:\anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "e:\anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "E:\anaconda3\Scripts\accelerate.exe\__main__.py", line 7, in <module>
  File "e:\anaconda3\lib\site-packages\accelerate\commands\accelerate_cli.py", line 43, in main
    args.func(args)
  File "e:\anaconda3\lib\site-packages\accelerate\commands\launch.py", line 837, in launch_command
    simple_launcher(args)
  File "e:\anaconda3\lib\site-packages\accelerate\commands\launch.py", line 354, in simple_launcher
    raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd)
subprocess.CalledProcessError: Command '['e:\\anaconda3\\python.exe', 'train_dreambooth.py', '--pretrained_model_name_or_path=CompVis/stable-diffusion-v1-4', '--instance_data_dir=./xyz dog', '--class_data_dir=./output/class_images', '--output_dir=./output', '--instance_prompt=a photo of xyz dog', '--class_prompt=a photo of a dog', '--resolution=512', '--train_batch_size=1', '--gradient_accumulation_steps=1', '--learning_rate=5e-6', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--num_class_images=200', '--max_train_steps=20']' returned non-zero exit status 1.

Reproduction

No response

Logs

No response

System Info

diffusers version: 0.7.0.dev0
Platform: Windows-10-10.0.22621-SP0
Python version: 3.7.6
PyTorch version (GPU?): 1.13.0 (True)
Huggingface_hub version: 0.10.1
Transformers version: 4.23.1
Using GPU in script?: no
Using distributed or parallel set-up in script?: no
Accelerate version: 0.12.0
Platform: Windows-10-10.0.22621-SP0
Python version: 3.7.6
Numpy version: 1.18.1
PyTorch version (GPU?): 1.13.0 (True)
Accelerate default config:
- compute_environment: LOCAL_MACHINE
- distributed_type: NO
- mixed_precision: bf16
- use_cpu: True
- num_processes: 1
- machine_rank: 0
- num_machines: 1
- main_process_ip: None
- main_process_port: None
- main_training_function: main
- deepspeed_config: {}
- fsdp_config: {}
- downcast_bf16: no

Update: removing safety_checker=None in train_dreambooth.py or/and handling it in the utils gives me an error when loading the checkpoint (after converting the training output for stable diffusion). What type of object is the safety_checker expecting?

Loading weights [9453d9a1] from D:\Downloads\stable-diffusion-webui\models\Stable-diffusion\training_output.ckpt
Error verifying pickled file from D:\Downloads\stable-diffusion-webui\models\Stable-diffusion\training_output.ckpt:
Traceback (most recent call last):
  File "D:\Downloads\stable-diffusion-webui\modules\safe.py", line 97, in load
    check_pt(filename)
  File "D:\Downloads\stable-diffusion-webui\modules\safe.py", line 77, in check_pt
    check_zip_filenames(filename, z.namelist())
  File "D:\Downloads\stable-diffusion-webui\modules\safe.py", line 69, in check_zip_filenames
    raise Exception(f"bad file inside {filename}: {name}")
Exception: bad file inside D:\Downloads\stable-diffusion-webui\models\Stable-diffusion\training_output.ckpt: training_output/data.pkl

The file may be malicious, so the program is not going to read it.
You can skip this check with --disable-safe-unpickle commandline argument.

Traceback (most recent call last):
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 275, in run_predict
    output = await app.blocks.process_api(
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 787, in process_api
    result = await self.call_function(fn_index, inputs, iterator)
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 694, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "D:\Downloads\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "D:\Downloads\stable-diffusion-webui\modules\ui.py", line 1736, in <lambda>
    fn=lambda value, k=k: run_settings_single(value, key=k),
  File "D:\Downloads\stable-diffusion-webui\modules\ui.py", line 1591, in run_settings_single
    opts.data_labels[key].onchange()
  File "D:\Downloads\stable-diffusion-webui\webui.py", line 40, in f
    res = func(*args, **kwargs)
  File "D:\Downloads\stable-diffusion-webui\webui.py", line 85, in <lambda>
    shared.opts.onchange("sd_model_checkpoint", wrap_queued_call(lambda: modules.sd_models.reload_model_weights(shared.sd_model)))
  File "D:\Downloads\stable-diffusion-webui\modules\sd_models.py", line 265, in reload_model_weights
    load_model_weights(sd_model, checkpoint_info)
  File "D:\Downloads\stable-diffusion-webui\modules\sd_models.py", line 169, in load_model_weights
    if "global_step" in pl_sd:
TypeError: argument of type 'NoneType' is not iterable

ShivamShrirao / diffusers