search

ZHO-ZHO-ZHO / ComfyUI-InstantID

Unofficial implementation of InstantID for ComfyUI
1.32k stars 76 forks source link

Make ControlNet loader optional if no image or controlnet_conditioning_scale is 0 - it can save VRAM #69

Open hortom opened 8 months ago

hortom commented 8 months ago

Hi, Is it possible to make ContorlNet optional and save VRAM? So, InstantID would be able to run stable on 12GB GPUs.

If controlnet_conditioning_scale is 0 or no connected pose_image_optional then ControlNet looks unnecessary.

If the above is correct, please amend your solution.

Cheers

cerarslan commented 8 months ago

Actually, I can run with 3060 12gb but not always. It is not stable

hortom commented 8 months ago

Actually, I can run with 3060 12gb but not always. It is not stable

Yes, same here with 4070. Let me add "stable" to my first comment.

hortom commented 8 months ago

I have a solution for memory optimisation. Actually, I have 2 levels. My hunch is that they are not entirely stable. Eg. you can't switch from one to another or disable them if not needed after a run. This needs extra work. I will try to share via a PR. I can run InstantID on my 4070 (12GB VRAM) many times.

Level 1 This is a faster solution. It uses enable_model_cpu_offload() VRAM usage during calculation: 85-88% Speed: 1.25 it/s Pros: relatively fast Cons: after 5-8 runs, OOM error is thrown, so need extra work to prevent memory leaking or better caching. A more clever dev can figure this out, I am sure.

Level 2 This is a slower solution. It uses enable_sequential_cpu_offload() VRAM usage during calculation:15-17% Speed: 2.35 s/it (2-3x slower) Pros: I can run this multiple times without any issues. Cons: Still, need VRAM to load the models, for this needs around 10 GB.

Please note that these are hack-ish solutions as these functions shouldn't be called more than once on a pipe, AFAIK. So, I had to add extra checking to detect if they were called (enable_sequential_cpu_offload()) or if possible, revert its effect (enable_model_cpu_offload()).

hortom commented 8 months ago

PR is here: https://github.com/ZHO-ZHO-ZHO/ComfyUI-InstantID/pull/87

cerarslan commented 8 months ago

Vaov thank you so much mate 🙏

shakenbake604 commented 8 months ago

To condense the text regarding making ControlNet optional to save VRAM for stable InstantID on 12GB GPUs, here's a summary based on the search results:

  1. There's a discussion on whether ControlNet can be made optional to reduce VRAM usage, which would allow InstantID to run more efficiently on 12GB GPUs. If certain conditions are met, such as controlnet_conditioning_scale being set to 0 or no pose image connected, ControlNet might be unnecessary, suggesting a potential for optimization [1].

  2. InstantID uses two models in its UI, with the ipadapter model needing to be set up first before the ControlNet model. This sequence is important because ControlNet takes the output from the ipadapter model. Adjustments in the settings could potentially impact VRAM usage [2].

  3. A low VRAM mode for ControlNet is available, which can help save memory and potentially increase batch size by modifying configuration files. This mode is experimental and may not be compatible with all GPUs, but it's an option for reducing VRAM consumption [3].

  4. Users have reported needing to switch to low VRAM mode after updates, indicating that there may be ongoing issues with VRAM usage that could be addressed in future updates. This suggests that there is a recognized need for better VRAM management within the community [4].

In essence, there are ongoing discussions and potential solutions for making ControlNet optional or more VRAM-efficient for users with 12GB GPUs wanting to run InstantID stably.