Model load for 21 hours

Priyanshu88 commented 3 months ago

Expected Behavior

The expected model should not take this much time.

Actual Behavior

It showing me that model loading will require more than 21 hours.

Steps to Reproduce

e_workflow.json

Debug Logs

[INFO] ComfyUI-Impact-Pack: SAM model loaded.
got prompt
model_type EPS
Using split attention in VAE
Using split attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
Requested to load AutoencoderKL
Loading 1 new model
Requested to load BaseModel
Loading 1 new model
  7%|███▋                                                    | 1/15 [1:31:05<21:15:15, 5465.36s/it][Impact Pack] Wildcards loading done.

Other

No response

ltdrdata commented 3 months ago

To identify this issue, you need to clearly specify the environment you're using. The phenomenon appears as if you're loading the model through a very slow network drive.

Priyanshu88 commented 3 months ago

Thanks for responding! I refreshed and reloaded and got the below error:

got prompt
model_type EPS
Using split attention in VAE
Using split attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
Requested to load AutoencoderKL
Loading 1 new model
!!! Exception during processing!!! [enforce fail at alloc_cpu.cpp:114] data. DefaultCPUAllocator: not enough memory: you tried to allocate 20423839744 bytes.
Traceback (most recent call last):
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 294, in encode
    t = vae.encode(pixels[:,:,:,:3])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 361, in encode
    samples[x:x+batch_number] = self.first_stage_model.encode(pixels_in).to(self.output_device).float()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\models\autoencoder.py", line 179, in encode
    z = self.encoder(x)
        ^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 531, in forward
    h = self.mid.attn_1(h)
        ^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 287, in forward
    h_ = self.optimized_attention(q, k, v)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 206, in normal_attention
    r1 = slice_attention(q, k, v)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 180, in slice_attention
    s1 = torch.bmm(q[:, i:end], k) * scale
         ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
RuntimeError: [enforce fail at alloc_cpu.cpp:114] data. DefaultCPUAllocator: not enough memory: you tried to allocate 20423839744 bytes.

Prompt executed in 1716.65 seconds

Previously I correct this by removing the automatic paging file size, also providing the some value and got the further loading but now i am not getting model loaded.

Can you please help me resolve this, even though I just want to confirm that the approach of the workflow inpaints the top-wear or bottom-wear of an image. You response would be helpful. Thanks!

ltdrdata commented 3 months ago

Thanks for responding! I refreshed and reloaded and got the below error:

got prompt
model_type EPS
Using split attention in VAE
Using split attention in VAE
Requested to load SD1ClipModel
Loading 1 new model
Requested to load AutoencoderKL
Loading 1 new model
!!! Exception during processing!!! [enforce fail at alloc_cpu.cpp:114] data. DefaultCPUAllocator: not enough memory: you tried to allocate 20423839744 bytes.
Traceback (most recent call last):
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\nodes.py", line 294, in encode
    t = vae.encode(pixels[:,:,:,:3])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\sd.py", line 361, in encode
    samples[x:x+batch_number] = self.first_stage_model.encode(pixels_in).to(self.output_device).float()
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\models\autoencoder.py", line 179, in encode
    z = self.encoder(x)
        ^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 531, in forward
    h = self.mid.attn_1(h)
        ^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 287, in forward
    h_ = self.optimized_attention(q, k, v)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 206, in normal_attention
    r1 = slice_attention(q, k, v)
         ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\pande\Downloads\ComfyUI_windows_portable_nvidia_cu121_or_cpu\ComfyUI_windows_portable\ComfyUI\comfy\ldm\modules\diffusionmodules\model.py", line 180, in slice_attention
    s1 = torch.bmm(q[:, i:end], k) * scale
         ~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
RuntimeError: [enforce fail at alloc_cpu.cpp:114] data. DefaultCPUAllocator: not enough memory: you tried to allocate 20423839744 bytes.

Prompt executed in 1716.65 seconds

Previously I correct this by removing the automatic paging file size, also providing the some value and got the further loading but now i am not getting model loaded.

Can you please help me resolve this, even though I just want to confirm that the approach of the workflow inpaints the top-wear or bottom-wear of an image. You response would be helpful. Thanks!

What is hardware spec? RAM size, GPU, Disk (SSD? HDD?)

Priyanshu88 commented 3 months ago

Windows 11 RAM - 8 GB SSD - 512GB Running the CPU file since no GPU available.

Can you please confirm if the workflow is working correctly and performing image inpainting for the same case as I mentioned.

ltdrdata commented 3 months ago

Windows 11 RAM - 8 GB SSD - 512GB Running the CPU file since no GPU available.

Can you please confirm if the workflow is working correctly and performing image inpainting for the same case as I mentioned.

Check your RAM usage in Task Manager while running ComfyUI.

Priyanshu88 commented 3 months ago

And what should i expect from the check?

Windows 11 RAM - 8 GB SSD - 512GB Running the CPU file since no GPU available. Can you please confirm if the workflow is working correctly and performing image inpainting for the same case as I mentioned.

Check your RAM usage in Task Manager while running ComfyUI.

And what should i expect from the check?

ltdrdata commented 3 months ago

And what should i expect from the check?

Windows 11 RAM - 8 GB SSD - 512GB Running the CPU file since no GPU available. Can you please confirm if the workflow is working correctly and performing image inpainting for the same case as I mentioned.

Check your RAM usage in Task Manager while running ComfyUI.

And what should i expect from the check?

Your RAM appears to be insufficient to handle Stable Diffusion.

Just running Windows and launching ComfyUI is already pushing your memory to its limits. It's expected that paging will occur the moment you try to load a model.

Keep in mind that if massive paging occurs, your computer's performance will drop to a level where there's essentially nothing you can do except reboot the computer.

Priyanshu88 commented 3 months ago

Yes I am facing this problem. The browsers and applications become unresponsive and at the end I get the below error.

I shall assume that it will not work with my system configuration?

It would be nice if you can help me validating the workflow if possible, as it is for a submission. Thanks!

ltdrdata commented 3 months ago

Yes I am facing this problem. The browsers and applications become unresponsive and at the end I get the below error.

I shall assume that it will not work with my system configuration?

It would be nice if you can help me validating the workflow if possible, as it is for a submission. Thanks!

You need to upgrade your machine. There is no other way.

comfyanonymous / ComfyUI