search

LaurentMazare / diffusers-rs

An implementation of the diffusers api in Rust
Apache License 2.0
539 stars 55 forks source link

ControlNet support? #56

Open katopz opened 1 year ago

katopz commented 1 year ago

I think it would be nice to have ControlNet support. Not sure how hard it gonna take for this task. 🤔

zolrath commented 1 year ago

Would absolutely love to use ControlNet scribble here!

LaurentMazare commented 1 year ago

I just merged some very basic support for ControlNet via #68 . The weight file data/controlnet.safetensors can be obtained from the HuggingFace hub here.

Note that this is for the "canny" version of ControlNet and the edge detection part has to be done before submitting the input image (I plan on fixing this soonish). Here is how to run it:

cargo run --example controlnet --features clap -- --cpu all --prompt "disco dancer with colorful lights" --input-image vermeer_canny_edged.png

The input image vermeer_canny_edged.png looked like: vermeer_canny_edged

Generated image: vermeer-gen

katopz commented 1 year ago

Cool! I learn a lot from this feature PR. Thanks! I will find some free time to add other ControlNet capabilities.

LaurentMazare commented 1 year ago

Great that you find it helpful, I've also included the edge detection bit now (through an external crate) so there is no need to pre-process the original image anymore. There are also some specific instructions in the readme file.

otdavies commented 1 year ago

If I wanted to stack multiple controlnets together (depth, canny, normal) for example, how difficult would you estimate this to be to implement @LaurentMazare?