Which model to load when using stable-diffusion configs?

[wonjunior]

Which model model.ckpt should be used when using the stable diffusion configuration file (configs/stable-diffusion/v1-inference.yaml) to optimize a concept?

[kaneyxx]

Which model model.ckpt should be used when using the stable diffusion configuration file (configs/stable-diffusion/v1-inference.yaml) to optimize a concept?

Have you tried it? I have the same question.

[wonjunior]

This seems to be solved here. Loading the original CompVis Stable Diffusion model weights along with the inference config configs/stable-diffusion/v1-inference.yaml seems to be sufficient to perform an inversion with stable diffusion. I have yet to try that.

[kaneyxx]

Thanks for information! @wonjunior I'd like to try it this weekend.

[kaneyxx]

@wonjunior I've tried it yesterday with that stable diffusion original v1.4 checkpoint downloaded from huggingface. Training process is fine, but there are some error happened during evaluation.

Saving latest checkpoint...

Traceback (most recent call last):
  File "/data/fangyi/textual_inversion/main.py", line 808, in <module>
    trainer.test(model, data)
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 911, in test
    return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 685, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 954, in _test_impl
    results = self._run(model, ckpt_path=self.tested_ckpt_path)
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/trainer.py", line 1128, in _run
    verify_loop_configurations(self)
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 42, in verify_loop_configurations
    __verify_eval_loop_configuration(trainer, model, "test")
  File "/home/fangyi/miniconda3/envs/torch/lib/python3.10/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 186, in __verify_eval_loop_configuration
    raise MisconfigurationException(f"No `{loader_name}()` method defined to run `Trainer.{trainer_method}`.")
pytorch_lightning.utilities.exceptions.MisconfigurationException: No `test_dataloader()` method defined to run `Trainer.test`.

Looks something wrong in pytorch_lightning, any idea?

[rinongal]

For the stable diffusion version you need any of the compvis-based models, so 1.4 and 1.5 will both work fine, as will models tuned from those base models (e.g. protogen).

@kaneyxx There's just no test-set defined for the model so it crashes when training is done. If you want to avoid that crash you can look at the training config data portion and copy the "train:" block into a "test:" block.