search

exx8 / differential-diffusion

398 stars 23 forks source link

ip adapter and controlnet pipelines #19

Open loboere opened 8 months ago

loboere commented 8 months ago

This has a lot of potential if used together with ip adapter and controlnet However, the current pipeline is very old and does not support the new features. can you please update and add pipelines for ip adapter and controlnet

CesarERamosMedina commented 8 months ago

With the way the latents are calculated in diff diff, I am unsure whether this would work out of the box for controlnet and ip adapter pipelines. My early experiments just output noise, even after a large number of steps.

exx8 commented 8 months ago

Hi, Sorry for the delay and thanks for your interest in diff diff. I actually tested now the paper with ControlNet and it works! I haven't decided what is my support plan for this repo yet. As fields progress rapidly, I believe we need a lasting solution, especially considering that more papers expected to work with diff diff are expected to be released. I will try to speak with https://github.com/huggingface/diffusers 's maintainers.

berkanz commented 3 months ago

Hi, do you plan to share the code with controlnet implementation? I experienced also a rough noise problem similar to @CesarERamosMedina, especially with white map images.

exx8 commented 3 months ago

I'm considering sharing my original, non-optimized implementation with the public and asking for assistance from the Diffusers team. They've done an amazing job, particularly with the introduction of diff diff to SD3 and HunyuanDiT. I plan to email them to see if they're interested. Alternatively, you could open an issue in the Diffusers repository, and I'll try to help as well. https://github.com/huggingface/diffusers

Thank you for your interest in diff-diff!