search

huggingface / swift-coreml-diffusers

Swift app demonstrating Core ML Stable Diffusion
Apache License 2.0
2.54k stars 217 forks source link

Can you change the stable-diffusion Swift Package to use StableDiffusionControlNetPipeline instead of StableDiffusionPipeline? #41

Open yerx opened 1 year ago

yerx commented 1 year ago

The stable-diffusion Swift package uses the 'StableDiffusionPipeline' class. How can I change the pipeline to use the 'StableDiffusionControlNetPipeline'? I will be using the diffusion model runwaylml/stable-diffusion-v1-5 so I don't think I need to go through the step of converting the stable-diffusion-v1-5 model into a CoreML model.

pcuenca commented 1 year ago

Hi @yerx! You won't need to convert the stable-diffusion-v1-5 model, but you'd need to convert the ControlNet model(s) you'll be using, and then modify the pipeline code to use both the ControlNet model and the base one.

May I ask what use-case you have in mind, and what ControlNet model are you planning to use?

yerx commented 1 year ago

Hi @pcuenca , thanks for responding. I'm interested in using the Canny and Scribble ControlNet models.

artarchive commented 1 year ago

I'm really excited for this feature to be made available on the Mac! Is there a process to get ControlNet to (finally!) work for yerx, myself and anyone else on the Mac? <3 Thank you so much!

artarchive commented 1 year ago

I was overly excited to see something else on the SD space than on the Web UI, so that brought me here. Upon more digging, I found something shockingly good... "DrawThings:" https://apps.apple.com/us/app/draw-things-ai-generation/id6444050820

DT does everything that I'm watching on YouTube about SD. <3

I hope this addendum here to my comment I made yesterday clears my thoughts up, sorry for any confusion. Cheers and have a good one!

svenkevs commented 1 year ago

In light of the release of the controlnet models on HF by the CoreML team: https://huggingface.co/coreml/ControlNet-Models-For-Core-ML

Will official support for the controlnet pipeline be considered? Mochi diffusion has already implemented it if an example is needed.