very nice - thank you

[vladmandic]

added implementation to sdnext via commit https://github.com/vladmandic/automatic/commit/02394356e78b7202e855200dcda23ee652604394

do you have plans to add sd15 pipeline?

[exx8]

Wow thank you so much! So SD1.5 is actually supported by the SD2 pipeline. To change for the SD1.5 model you need to edit this: https://github.com/exx8/differential-diffusion/blob/c02c25ff75f8dd24f56cae9a0bb885714d00bbc1/SD2/run.py#L9 To this: pipe = StableDiffusionDiffImg2ImgPipeline.from_pretrained("runwayml/stable-diffusion-v1-5”, torch_dtype=torch.float16).to(device) I understand it might be a bit misleading that the folder is called “SD2” but today SD means at least 4 different families of architectures. More info:

1. You can of course use any other checkpoint which was fine-tuned from those base models (Stable Diffusion 1 and 2) with this pipeline.
2. It's expected that all other present diffusion models are supported as well by the framework (with a ported version). Including those who do not use U-Net architecture.

[vladmandic]

thanks!

i've already implemented pipeline switching, i'm not loading a model using your pipeline, i'm using whatever model user already has loaded and just instantiating your pipeline as needed to perform a task.

btw, note that some imports have changed in latest diffusers, you may want to update the code - for example:

from diffusers.utils.torch_utils import randn_tensor

[exx8]

thanks!

i've already implemented pipeline switching, i'm not loading a model using your pipeline, i'm using whatever model user already has loaded and just instantiating your pipeline as needed to perform a task.

btw, note that some imports have changed in latest diffusers, you may want to update the code - for example:

from diffusers.utils.torch_utils import randn_tensor

Thank you! The next step is integrating the paper with diffusers!

[vladmandic]

SD pipeline is a bit out-of-date compared to SDXL pipeline as its missing PEFT support, encode_prompt method, different image preprocess, support for non-PIL output (i prefer to have output_type="latent" output and postprocess separately), but got it working eventually - so both SD15 and SDXL are now implemented.

btw, differential-diffusion combines really well with a traditional depth models to auto-create mask.

[exx8]

SD pipeline is a bit out-of-date compared to SDXL pipeline as its missing PEFT support, encode_prompt method, different image preprocess, support for non-PIL output (i prefer to have output_type="latent" output and postprocess separately), but got it working eventually - so both SD15 and SDXL are now implemented.

btw, differential-diffusion combines really well with a traditional depth models to auto-create mask.

Thank you! Yeah you are right. Depth models tend to create good change maps. In the new manuscript, we show that you can even create from one depth map multiple of change maps by simple histogram shifts.

[vladmandic]

anyhow, i'm closing this issue as it was more fyi - but feel free to post if you have any feedback for me, i'll still get notifications.

[exx8]

Thank you for this issue. This issue has brought great joy for me! If you have any more comments or questions, please feel free to share them!