search

lllyasviel / ControlNet

Let us control diffusion models!
Apache License 2.0
28.97k stars 2.62k forks source link

Integration Challenge: Encoder Architectures in Control Net Input #599

Open 0x434D opened 7 months ago

0x434D commented 7 months ago

I'm looking to play around with the input for the control net. Basically, I want to hook up some encoder architectures in front of the input into the control net. By that, I mean running the conditions through an encoder before tossing them into the control net as conditions.

My issue is figuring out where to slot in this architecture in the code, especially making sure it gets trained alongside the control net during training.

Thanks a lot in advance for your help — I hope I've explained what I'm aiming for clearly enough.