search

lllyasviel / ControlNet-v1-1-nightly

Nightly release of ControlNet 1.1
4.48k stars 364 forks source link

Joint training of ControlNets with different conditions #41

Closed andreemic closed 1 year ago

andreemic commented 1 year ago

Hey! Thanks for all the amazing work on this project @lllyasviel.

We at Virtual Staging AI are experimenting with modifying ControlNet for virtual staging (adding furniture to empty rooms). Here's an example input/output pair using a 3D reconstruction model.

Screenshot 2023-04-02 at 13 13 00

With ControlNet we've achieved fairly good results by superimposing M-LSD lines on top of the original room image Obviously you can get powerful results by combining multiple conditions as described in the ControlNet article on Hugging Face but does it also make sense to jointly train ControlNets with different conditions?

For example, train with two conditions as input:

If so, is there code to train multiple ControlNets jointly? Or one ControlNet on multiple conditions?

Semolce commented 1 year ago

Diffusers have already enabled multi-control. I think you can train a multiple controlnets based on this script train_controlnet.py with a little modification, saying change the controlnet it use to multi controlnet and I think it will work fine. Here's an example of how to enable multi controlnetmulti-control

lllyasviel commented 1 year ago

Thanks for the issue. Multi-controlnet is a1111-only now.

hubble-bubble commented 10 months ago

Hey! Thanks for all the amazing work on this project @lllyasviel.

We at Virtual Staging AI are experimenting with modifying ControlNet for virtual staging (adding furniture to empty rooms). Here's an example input/output pair using a different model. Screenshot 2023-04-02 at 13 13 00

With ControlNet we've achieved fairly good results by superimposing M-LSD lines on top of the original room image Obviously you can get powerful results by combining multiple conditions as described in the ControlNet article on Hugging Face but does it also make sense to jointly train ControlNets with different conditions?

For example, train with two conditions as input:

  • raw image of a room and
  • depth map of the room So the model learns to keep the exact room pixels from the raw image and learns to insert furniture sensibly from the depth map.

If so, is there code to train multiple ControlNets jointly? Or one ControlNet on multiple conditions?

have you find the way how to train multiple ControlNets jointly？

geroldmeisinger commented 9 months ago

see here https://github.com/lllyasviel/ControlNet/discussions/30