[Bug] VRAM leak for PuLID in IpAdapter!

inspire-boy commented 6 months ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

PuLID Vram leak while switching input image.

Thanks for the update provided by @huchenlei . I then tested it in the 3090-24G environment and the problem still seemed to exist. This issue can be resolved by restarting the webui, But it's a fatal error for some Integrated application, it‘s also important for a widely used project like ControNet~

Steps to reproduce the problem

1.start webui 2.use model: sdxl_lightning/sdxl_lightning_4step.safetensors , following is my generation params: _portrait,cinematic,wolf ears,white hair Negative prompt: flaws in the eyes, flaws in the face, flaws, lowres, non-HDRi, low quality, worst quality,artifacts noise, text, watermark, glitch, deformed, mutated, ugly, disfigured, hands, low resolution, partially rendered objects, deformed or partially rendered eyes, deformed, deformed eyeballs, cross-eyed,blurry Steps: 4, Sampler: DPM++ SDE, Schedule type: SGM Uniform, CFG scale: 1.2, Seed: 2461588618, Size: 768x1024, Model: sdxl_lightning_4step, VAE hash: 63aeecb90f, VAE: sdxl_lightning_4step.vae.safetensors, Clip skip: 2, ControlNet 0: "Module: ip-adapter-auto, Model: ip-adapter_pulid_sdxlfp16 [d86d05ea], Weight: 0.8, Resize Mode: Crop and Resize, Processor Res: 512, Threshold A: 0.5, Threshold B: 0.5, Guidance Start: 0.0, Guidance End: 1.0, Pixel Perfect: False, Control Mode: Balanced", Eta: 0.2, Version: v1.9.3 3.use PuLID Module: ip-adapter-auto, Model: ip-adapter_pulid_sdxl_fp16 , and an input image, Make a generation. 4.finished a run, change a diffrent input image. make a generation again. Cycle this step. 13 times(may be more). 5.Vram OOM. 微信截图_20240519225248

What should have happened?

When a run over. vram should drop to the level of the last finish. but the base Vram is increased each time. OOM is displayed several times runs. Switching between different input images can be reproduced

Commit where the problem happens

webui: #https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/1c0a0c4c26f78c32095ebc7f8af82f5c04fca8c0 controlnet: #https://github.com/Mikubill/sd-webui-controlnet/commit/681de2c4f0298001195b1133aab9d0d429102148

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

bash webui.sh -f  --api  --no-half-vae --xformers --medvram --disable-nan-check

List of enabled extensions

微信截图_20240519231840

Console logs

return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
    torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 26.00 MiB (GPU 0; 23.69 GiB total capacity; 21.85 GiB already allocated; 26.94 MiB free; 23.32 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

Additional information

No response

huchenlei commented 6 months ago

The VRAM leak comes from facexlib and evaclip. Now in this extension we are doing the same thing as in the PuLID main repo to free memory.

At least with my local testing, the VRAM leak issue is fixed. Do you also experience VRAM leak when using diffuser impl of PuLID?

inspire-boy commented 6 months ago

The VRAM leak comes from facexlib and evaclip. Now in this extension we are doing the same thing as in the PuLID main repo to free memory.

At least with my local testing, the VRAM leak issue is fixed. Do you also experience VRAM leak when using diffuser impl of PuLID?

I tested Diffusers and PuLID_ComfyUI. There both are no leak problems.

huchenlei commented 6 months ago

Well, maybe A1111 just needs a better VRAM management system to handle this. I probably won't be able to fix this now as I cannot reproduce it locally.

inspire-boy commented 6 months ago

Well, maybe A1111 just needs a better VRAM management system to handle this. I probably won't be able to fix this now as I cannot reproduce it locally.

Thanks your reply. This may be a problem with the webui, but I'm confused that other ControlNet models in webui I've used don't have this problem. Such as Canny/Brightness/Depth/qr_monster and those model have huge size.

akk-123 commented 6 months ago

@huchenlei can you support pulid on webui-forge?

lonmusk commented 5 months ago

VRAM leak for PuLID in IpAdapter! I'm having the same issue，And when I updated the controlnet to the latest version according to the solution you submitted by the author, and the code was also reviewed, there was still such an error.

lonmusk commented 5 months ago

VRAM leak for PuLID in IpAdapter! I'm having the same issue，And when I updated the controlnet to the latest version according to the solution you submitted by the author, and the code was also reviewed, there was still such an error.

When I use pulid, my video memory is getting less and less, and I still have to reboot to use it

AbelianKraun commented 5 months ago

Hi!

Is there any update on the issue?

I can confirm that with all the last updates, I still have the memory leak.

Does anyone have at least some workarounds?

Thank you.

Mikubill / sd-webui-controlnet