Hi! Awesome job! This is a non-issue but discussion. I think your idea in the paper is essentially two things, more connections (or control just in time) and smaller control nets. I saw the team of diffusers did smaller control nets when they train a control net of SDXL here, and I saw stability uses a LoRA to be a control net here.
I haven't taken a closer look in the code of these two, but these make me wonder that probably we can take your smaller control nets as a LoRA that not only modify the behaviour of SD but also extends it, i.e. seeing the control images.
I also wonder whether we can just use LoRA to do your smaller control nets, then probably we can achieve even faster training and even smaller weights. I guess we can replace you networks with LoRAs while keeping your connections between generative encoders and control encoders.
Hi! Awesome job! This is a non-issue but discussion. I think your idea in the paper is essentially two things, more connections (or control just in time) and smaller control nets. I saw the team of diffusers did smaller control nets when they train a control net of SDXL here, and I saw stability uses a LoRA to be a control net here.
I haven't taken a closer look in the code of these two, but these make me wonder that probably we can take your smaller control nets as a LoRA that not only modify the behaviour of SD but also extends it, i.e. seeing the control images.
I also wonder whether we can just use LoRA to do your smaller control nets, then probably we can achieve even faster training and even smaller weights. I guess we can replace you networks with LoRAs while keeping your connections between generative encoders and control encoders.