when Using SDXL-control-lora with 6GB VRAM, "Ran out of memory"

[youyegit]

I have tested SDXL in comfyui with RTX2060 6G， when I use "sai_xl_canny_128lora.safetensors" or "sai_xl_depth_128lora.safetensors", it will show "Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding." but when I use "diffusers_xl_canny_small.safetensors" or "diffusers_xl_depth_small.safetensors", it works well.

when I use clip_vison, it works well too.

So the problem may come from comfyui's nodes. This maybe a bug.

Thanks for the great job and hope comfyui better.

[youyegit]

and in comfyui user interface , it show the error:

Error occurred when executing VAEDecode:

Allocation on device 0 would exceed allowed memory. (out of memory) Currently allocated : 4.58 GiB Requested : 256.00 MiB Device limit : 6.00 GiB Free (according to CUDA): 0 bytes PyTorch limit (set by user-supplied memory fraction) : 17179869184.00 GiB

File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "E:\Z_comfyui_2023-10-17\ComfyUI\nodes.py", line 267, in decode return (vae.decode(samples["samples"]), ) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 206, in decode pixel_samples = self.decodetiled(samples_in) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 173, in decodetiled comfy.utils.tiled_scale(samples, decode_fn, tile_x 2, tile_y // 2, overlap, upscale_amount = 8, pbar = pbar) + File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\utils.py", line 395, in tiled_scale ps = function(s_in).cpu() File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 170, in decode_fn = lambda a: (self.first_stage_model.decode(a.to(self.vae_dtype).to(self.device)) + 1.0).float() File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\models\autoencoder.py", line 94, in decode dec = self.decoder(z) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 726, in forward h = self.up[i_level].upsample(h) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 72, in forward x = self.conv(x) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride,

[youyegit]

This the log of comfyui when using "sai_xl_canny_128lora.safetensors" :

Python 3.10.6 pip 23.1.2 from E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\pip (python 3.10) ** ComfyUI start up time: 2023-10-18 13:03:59.815647

Prestartup times for custom nodes: 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\ComfyUI-Manager

Total VRAM 6144 MB, total RAM 32489 MB xformers version: 0.0.20 Forcing FP16. Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 2060 : cudaMallocAsync VAE dtype: torch.float32 disabling upcasting of attention Using xformers cross attention Setting temp directory to: E:\Z_comfyui_2023-10-17\temp\temp Adding extra search path checkpoints E:/AI/SD/sd-webui/stable-diffusion-webui/models/Stable-diffusion Adding extra search path configs E:/AI/SD/sd-webui/stable-diffusion-webui/models/Stable-diffusion Adding extra search path vae E:/AI/SD/sd-webui/stable-diffusion-webui/models/VAE Adding extra search path loras E:/AI/SD/sd-webui/stable-diffusion-webui/models/Lora Adding extra search path upscale_models E:/AI/SD/sd-webui/stable-diffusion-webui/models/ESRGAN Adding extra search path upscale_models E:/AI/SD/sd-webui/stable-diffusion-webui/models/SwinIR Adding extra search path embeddings E:/AI/SD/sd-webui/stable-diffusion-webui/embeddings Adding extra search path hypernetworks E:/AI/SD/sd-webui/stable-diffusion-webui/models/hypernetworks Adding extra search path controlnet E:/AI/SD/sd-webui/stable-diffusion-webui/models/ControlNet Adding extra search path checkpoints ../extra_model/models/checkpoints Adding extra search path configs ../extra_model/models/configs Adding extra search path vae ../extra_model/models/vae Adding extra search path loras ../extra_model/models/loras Adding extra search path clip_vision ../extra_model/models/clip_vision Adding extra search path controlnet ../extra_model/models/controlnet Adding extra search path custom_nodes ../extra_model/custom_nodes

Loading: ComfyUI-Manager (V0.30.3) ComfyUI Revision: 1479 [f8032cdf] Registered sys.path: ['E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\init.py', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_pycocotools', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_oneformer', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mmpkg', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_midas_repo', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_detectron2', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src', 'E:\Z_comfyui_2023-10-17\ComfyUI\comfy', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\git\ext\gitdb', 'E:\Z_comfyui_2023-10-17\ComfyUI', 'E:\Z_comfyui_2023-10-17\python_embeded\python310.zip', 'E:\Z_comfyui_2023-10-17\python_embeded\DLLs', 'E:\Z_comfyui_2023-10-17\python_embeded\lib', 'E:\Z_comfyui_2023-10-17\python_embeded', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\win32', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\win32\lib', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\Pythonwin', '../..'] Registered sys.path: ['E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\init.py', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_pycocotools', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_oneformer', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mmpkg', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_midas_repo', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_detectron2', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\init.py', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_pycocotools', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_oneformer', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_mmpkg', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_midas_repo', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\custom_detectron2', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src\controlnet_aux', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux\src', 'E:\Z_comfyui_2023-10-17\ComfyUI\comfy', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\git\ext\gitdb', 'E:\Z_comfyui_2023-10-17\ComfyUI', 'E:\Z_comfyui_2023-10-17\python_embeded\python310.zip', 'E:\Z_comfyui_2023-10-17\python_embeded\DLLs', 'E:\Z_comfyui_2023-10-17\python_embeded\lib', 'E:\Z_comfyui_2023-10-17\python_embeded', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\win32', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\win32\lib', 'E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\Pythonwin', '../..', 'E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\tdxh_node_comfyui'] Total VRAM 6144 MB, total RAM 32489 MB xformers version: 0.0.20 Forcing FP16. Set vram state to: NORMAL_VRAM Device: cuda:0 NVIDIA GeForce RTX 2060 : cudaMallocAsync VAE dtype: torch.float32 WARNING: Ignoring invalid distribution -pencv-python (e:\z_comfyui_2023-10-17\python_embeded\lib\site-packages) WAS Node Suite: OpenCV Python FFMPEG support is enabled WAS Node Suite: ffmpeg_bin_path is set to: \ffmpeg-6.0-essentials_build\bin WARNING: Ignoring invalid distribution -pencv-python (e:\z_comfyui_2023-10-17\python_embeded\lib\site-packages) WAS Node Suite: Finished. Loaded 193 nodes successfully.

"Believe you deserve it and the universe will serve it." - Unknown

Import times for custom nodes: 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\reference_only.py 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\ControlNet-LLLite-ComfyUI 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\sdxl_prompt_styler 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\ComfyUI-Custom-Scripts 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\Derfuu_ComfyUI_ModdedNodes 0.0 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\tdxh_node_comfyui 0.1 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\comfyui_controlnet_aux 0.4 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\ComfyUI-Manager 1.8 seconds: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\was_node_suite_comfyui

Setting output directory to: E:\Z_comfyui_2023-10-17\output Setting input directory to: E:\Z_comfyui_2023-10-17\input Starting server

To see the GUI go to: http://0.0.0.0:8188/ FETCH DATA from: E:\Z_comfyui_2023-10-17\ComfyUI\custom_nodes\ComfyUI-Manager\extension-node-map.json got prompt style: Default (Slightly Cinematic) text_positive: school gate text_negative: text_positive_styled: cinematic still school gate . emotional, harmonious, vignette, highly detailed, high budget, bokeh, cinemascope, moody, epic, gorgeous, film grain, grainy text_negative_styled: anime, cartoon, graphic, text, painting, crayon, graphite, abstract, glitch, deformed, mutated, ugly, disfigured model_type EPS adm 2560 making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... missing {'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'} loading new making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... Reshaping decoder.mid.attn_1.k.weight for SD format Reshaping decoder.mid.attn_1.proj_out.weight for SD format Reshaping decoder.mid.attn_1.q.weight for SD format Reshaping decoder.mid.attn_1.v.weight for SD format Reshaping encoder.mid.attn_1.k.weight for SD format Reshaping encoder.mid.attn_1.proj_out.weight for SD format Reshaping encoder.mid.attn_1.q.weight for SD format Reshaping encoder.mid.attn_1.v.weight for SD format model_type EPS adm 2816 making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... Working with z of shape (1, 4, 32, 32) = 4096 dimensions. making attention of type 'vanilla-xformers' with 512 in_channels building MemoryEfficientAttnBlock with 512 in_channels... missing {'cond_stage_model.clip_l.logit_scale', 'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_g.transformer.text_model.embeddings.position_ids'} loading new loading new loading in lowvram mode 3017.959864616394 4%|███▌ | 1/25 [00:05<02:17, 5.71s/ 8%|██████▌ | 2/25 [00:10<01:58, 5.1 12%|█████████▌ | 3/25 [00:15<01:49, 16%|█████████████▌ | 4/25 [00:19<01:4 20%|████████████████▌ | 5/25 [00:24<0 24%|███████████████████▌ | 6/25 [00:2 28%|███████████████████████▌ | 7/25 [ 32%|██████████████████████████▌ | 8/2 36%|█████████████████████████████▌ | 40%|████████████████████████████████▌ 44%|████████████████████████████████████ 48%|███████████████████████████████████████▌ 52%|██████████████████████████████████████████▌ 56%|█████████████████████████████████████████████▌ 60%|█████████████████████████████████████████████████▌ 64%|████████████████████████████████████████████████████▌ 68%|███████████████████████████████████████████████████████▌ 72%|█████████████████████████████████████████████████████████ 76%|█████████████████████████████████████████████████████████ 80%|█████████████████████████████████████████████████████████ 84%|█████████████████████████████████████████████████████████ 88%|█████████████████████████████████████████████████████████ 92%|█████████████████████████████████████████████████████████ 96%|█████████████████████████████████████████████████████████ 100%|█████████████████████████████████████████████████████████ 100%|█████████████████████████████████████████████████████████ █████████████████████████| 25/25 [01:50<00:00, 4.42s/it] loading new loading in lowvram mode 880.3931360244751 20%|████████████████▌ | 1/5 [00:05<0 40%|█████████████████████████████████▌ 60%|██████████████████████████████████████████████████▌ 80%|█████████████████████████████████████████████████████████ 100%|█████████████████████████████████████████████████████████ 100%|█████████████████████████████████████████████████████████ ███████████████████████████| 5/5 [00:23<00:00, 4.66s/it] Warning: Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding. !!! Exception during processing !!! Traceback (most recent call last): File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 203, in decode pixel_samples[x:x+batch_number] = torch.clamp((self.first_stage_model.decode(samples) + 1.0) / 2.0, min=0.0, max=1.0).cpu().float() File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\models\autoencoder.py", line 94, in decode dec = self.decoder(z) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 726, in forward h = self.up[i_level].upsample(h) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 72, in forward x = self.conv(x) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, torch.cuda.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory) Currently allocated : 4.27 GiB Requested : 1012.00 MiB Device limit : 6.00 GiB Free (according to CUDA): 0 bytes PyTorch limit (set by user-supplied memory fraction) : 17179869184.00 GiB

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\Z_comfyui_2023-10-17\ComfyUI\execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(slice_dict(input_data_all, i))) File "E:\Z_comfyui_2023-10-17\ComfyUI\nodes.py", line 267, in decode return (vae.decode(samples["samples"]), ) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 206, in decode pixel_samples = self.decodetiled(samples_in) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 173, in decodetiled comfy.utils.tiled_scale(samples, decode_fn, tile_x 2, tile_y // 2, overlap, upscale_amount = 8, pbar = pbar) + File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\utils.py", line 395, in tiled_scale ps = function(s_in).cpu() File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\sd.py", line 170, in decode_fn = lambda a: (self.first_stage_model.decode(a.to(self.vae_dtype).to(self.device)) + 1.0).float() File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\models\autoencoder.py", line 94, in decode dec = self.decoder(z) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 726, in forward h = self.up[i_level].upsample(h) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "E:\Z_comfyui_2023-10-17\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 72, in forward x = self.conv(x) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward return self._conv_forward(input, self.weight, self.bias) File "E:\Z_comfyui_2023-10-17\python_embeded\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward return F.conv2d(input, weight, bias, self.stride, torch.cuda.OutOfMemoryError: Allocation on device 0 would exceed allowed memory. (out of memory) Currently allocated : 4.58 GiB Requested : 256.00 MiB Device limit : 6.00 GiB Free (according to CUDA): 0 bytes PyTorch limit (set by user-supplied memory fraction) : 17179869184.00 GiB

Prompt executed in 203.19 seconds

[comfyanonymous]

Try temporarily removing all your custom nodes and see if you still have the issue.

[youyegit]

I find this problem when I use Fooocus which is based on comfyUI , and Fooocus fix it just now , maybe it can help: https://github.com/lllyasviel/Fooocus/issues/700

[youyegit]

and I test "sai_xl_sketch_128lora.safetensors" and "sai_xl_recolor_128lora.safetensors", these two control-loras work well.

[comfyanonymous]

Should be fixed now.

[youyegit]

It is fixed now ! awsome !

[ZSSsama]

I get this error when I'm using the XL model: EOFError: Ran out of input, what's going on