search

comfyanonymous / ComfyUI

The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface.
https://www.comfy.org/
GNU General Public License v3.0
49.69k stars 5.23k forks source link

Segmentation fault (core dumped) #2458

Open TheManOfficial opened 8 months ago

TheManOfficial commented 8 months ago

This is not identical to the other post with the same title.

I get this error when running any stable diffusion XL model, and a similar (but shorter) error when I run any 1.0 model. It throws the error directly after loading the model.

Error I get for all XL models: Very long, uploaded to pastebin.

Error I get for all 1.5/1.0 models:

missing {'cond_stage_model.clip_l.text_projection', 'cond_stage_model.clip_l.logit_scale'}
left over keys: dict_keys(['cond_stage_model.clip_l.transformer.text_model.embeddings.position_ids'])

PC Specs: OS: RedHat Enterprise Linux 9.2 Python Version: 3.9.18 CPU: Ryzen 5 3600 GPU: RX6600XT ROCm Version: 6.0

Installed Libraries:

aiohttp             3.9.1
aiosignal           1.3.1
argcomplete         1.12.0
async-timeout       4.0.3
attrs               23.2.0
Brlapi              0.8.2
chardet             4.0.0
cockpit             300.3
cupshelpers         1.0
dasbus              1.4
dbus-python         1.2.18
distro              1.5.0
einops              0.7.0
file-magic          0.4.0
filelock            3.9.0
frozenlist          1.4.1
fsspec              2023.12.2
gpg                 1.15.1
huggingface-hub     0.20.1
idna                2.10
Jinja2              3.1.2
libcomps            0.1.18
lxml                4.6.5
MarkupSafe          2.1.3
mpmath              1.3.0
multidict           6.0.4
networkx            3.0
nftables            0.1
numpy               1.24.1
packaging           23.2
perf                0.1
pexpect             4.8.0
Pillow              9.3.0
pip                 21.2.3
psutil              5.8.0
ptyprocess          0.6.0
pycairo             1.20.1
pycups              2.0.1
pycurl              7.43.0.6
pyenchant           3.2.0
PyGObject           3.40.1
pyparsing           2.4.7
PySocks             1.7.1
python-dateutil     2.8.1
python-linux-procfs 0.7.0
pytorch-triton-rocm 2.1.0
pyudev              0.22.0
pyxdg               0.27
PyYAML              5.4.1
regex               2023.12.25
requests            2.25.1
rpm                 4.16.1.3
safetensors         0.4.1
scipy               1.11.4
selinux             3.5
sepolicy            3.5
setools             4.4.3
setroubleshoot      3.3.32
setuptools          53.0.0
six                 1.15.0
sos                 4.6.0
sympy               1.12
systemd-python      234
tokenizers          0.15.0
torch               2.1.2+rocm5.6
torchaudio          2.1.2+rocm5.6
torchsde            0.2.6
torchvision         0.16.2+rocm5.6
tqdm                4.66.1
trampoline          0.1.2
transformers        4.36.2
triton              2.1.0
typing_extensions   4.4.0
urllib3             1.26.5
xformers            0.0.23.post1
yarl                1.9.4

Steps to reproduce:

  1. Install ComfyUI following the manual installation guide.
  2. Add any stable diffusion model and it's vae
  3. Start ComfyUI and attempt to generate any image through the UI

Full log (excluding above errors):

Total VRAM 8176 MB, total RAM 31748 MB
WARNING[XFORMERS]: xFormers can't load C++/CUDA extensions. xFormers was built for:
    PyTorch 2.1.2+cu121 with CUDA 1201 (you have 2.1.2+rocm5.6)
    Python  3.9.18 (you have 3.9.18)
  Please reinstall xformers (see https://github.com/facebookresearch/xformers#installing-xformers)
  Memory-efficient attention, SwiGLU, sparse and more won't be available.
  Set XFORMERS_MORE_DETAILS=1 for more details
xformers version: 0.0.23.post1
Set vram state to: NORMAL_VRAM
Device: cuda:0 AMD Radeon RX 6600 XT : native
VAE dtype: torch.float32
Using sub quadratic optimization for cross attention, if you have memory or speed issues try using: --use-split-cross-attention
Starting server

To see the GUI go to: http://127.0.0.1:8188
got prompt
model_type EPS
adm 0
Using split attention in VAE
Working with z of shape (1, 4, 32, 32) = 4096 dimensions.
Using split attention in VAE
[Long error]
Requested to load SD1ClipModel
Loading 1 new model
Segmentation fault (core dumped)

I've tried all of the relevant fixes in issue #1142 but none of them proved helpful.

Jinchu commented 7 months ago

I'm having exactly the same problem.

mdehaussy commented 7 months ago

Got same issue and resolved it by using those options (Got a Zephyrus G14 with RX6800S) : HSA_OVERRIDE_GFX_VERSION=10.3.0 HIP_VISIBLE_DEVICES=0 ROCR_VISIBLE_DEVICES=0 PYTORCH_HIP_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:512

maxwelljens commented 6 months ago

Got same issue and resolved it by using those options (Got a Zephyrus G14 with RX6800S) : HSA_OVERRIDE_GFX_VERSION=10.3.0 HIP_VISIBLE_DEVICES=0 ROCR_VISIBLE_DEVICES=0 PYTORCH_HIP_ALLOC_CONF=garbage_collection_threshold:0.6,max_split_size_mb:512

This fixed the issue for me with my RX 6700 XT.

grigio commented 1 month ago

I tried with --lowvram and the enviroment variables but I get with SDXL:

Error occurred when executing KSampler:

maximum recursion depth exceeded while calling a Python object

File "/root/ComfyUI/execution.py", line 151, in recursive_execute

comfyui-rocm  | ** ComfyUI startup time: 2024-07-12 18:40:15.371113
comfyui-rocm  | ** Platform: Linux
comfyui-rocm  | ** Python version: 3.10.14 (main, Mar 21 2024, 16:45:28) [GCC]
comfyui-rocm  | ** Python executable: /usr/bin/python3
comfyui-rocm  | ** ComfyUI Path: /root/ComfyUI
comfyui-rocm  | ** Log path: /root/comfyui.log
comfyui-rocm  | 
comfyui-rocm  | Prestartup times for custom nodes:
comfyui-rocm  |    0.4 seconds: /root/ComfyUI/custom_nodes/ComfyUI-Manager
comfyui-rocm  | 
comfyui-rocm  | Total VRAM 512 MB, total RAM 47379 MB
comfyui-rocm  | pytorch version: 2.1.2+rocm6.1.3
comfyui-rocm  | Set vram state to: LOW_VRAM
comfyui-rocm  | Device: cuda:0 AMD Radeon Graphics : native
comfyui-rocm  | Using split optimization for cross attention
comfyui-rocm  | ### Loading: ComfyUI-Manager (V2.46.3)
comfyui-rocm  | ### ComfyUI Revision: 1 [29c2e267] | Released on '2024-07-12'

comfyui-rocm  | got prompt
comfyui-rocm  | model_type EPS
comfyui-rocm  | Using split attention in VAE
comfyui-rocm  | Using split attention in VAE
comfyui-rocm  | Requested to load SDXLClipModel
comfyui-rocm  | Loading 1 new model
comfyui-rocm  | Requested to load SDXL
comfyui-rocm  | Loading 1 new model
comfyui-rocm  | loading in lowvram mode 64.0
  0%|          | 0/5 [00:00<?, ?it/s]/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py:599: UserWarning: Should have ta>=t0 but got ta=7.727652072906494 and t0=14.61464.
comfyui-rocm  |   warnings.warn(f"Should have ta>=t0 but got ta={ta} and t0={self._start}.")
  0%|          | 0/5 [00:09<?, ?it/s]
comfyui-rocm  | !!! Exception during processing!!! maximum recursion depth exceeded while calling a Python object
comfyui-rocm  | Traceback (most recent call last):
comfyui-rocm  |   File "/root/ComfyUI/execution.py", line 151, in recursive_execute
comfyui-rocm  |     output_data, output_ui = get_output_data(obj, input_data_all)
comfyui-rocm  |   File "/root/ComfyUI/execution.py", line 81, in get_output_data
comfyui-rocm  |     return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
comfyui-rocm  |   File "/root/ComfyUI/execution.py", line 74, in map_node_over_list
comfyui-rocm  |     results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
comfyui-rocm  |   File "/root/ComfyUI/nodes.py", line 1373, in sample
comfyui-rocm  |     return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
comfyui-rocm  |   File "/root/ComfyUI/nodes.py", line 1343, in common_ksampler
comfyui-rocm  |     samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
comfyui-rocm  |   File "/root/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 410, in motion_sample
comfyui-rocm  |     return orig_comfy_sample(model, noise, *args, **kwargs)
comfyui-rocm  |   File "/root/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 22, in informative_sample
comfyui-rocm  |     raise e
comfyui-rocm  |   File "/root/ComfyUI/custom_nodes/ComfyUI-Impact-Pack/modules/impact/sample_error_enhancer.py", line 9, in informative_sample
comfyui-rocm  |     return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
comfyui-rocm  |   File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
comfyui-rocm  |     return orig_comfy_sample(model, *args, **kwargs)
comfyui-rocm  |   File "/root/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/utils.py", line 112, in uncond_multiplier_check_cn_sample
comfyui-rocm  |     return orig_comfy_sample(model, *args, **kwargs)
comfyui-rocm  |   File "/root/ComfyUI/comfy/sample.py", line 43, in sample
comfyui-rocm  |     samples = sampler.sample(noise, positive, negative, cfg=cfg, latent_image=latent_image, start_step=start_step, last_step=last_step, force_full_denoise=force_full_denoise, denoise_mask=noise_mask, sigmas=sigmas, callback=callback, disable_pbar=disable_pbar, seed=seed)
comfyui-rocm  |   File "/root/ComfyUI/comfy/samplers.py", line 801, in sample
comfyui-rocm  |     return sample(self.model, noise, positive, negative, cfg, self.device, sampler, sigmas, self.model_options, latent_image=latent_image, denoise_mask=denoise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
comfyui-rocm  |   File "/root/ComfyUI/comfy/samplers.py", line 703, in sample
comfyui-rocm  |     return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
comfyui-rocm  |   File "/root/ComfyUI/comfy/samplers.py", line 690, in sample
comfyui-rocm  |     output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
comfyui-rocm  |   File "/root/ComfyUI/comfy/samplers.py", line 669, in inner_sample
comfyui-rocm  |     samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
comfyui-rocm  |   File "/root/ComfyUI/comfy/samplers.py", line 574, in sample
comfyui-rocm  |     samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
comfyui-rocm  |   File "/usr/lib64/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
comfyui-rocm  |     return func(*args, **kwargs)
comfyui-rocm  |   File "/root/ComfyUI/comfy/k_diffusion/sampling.py", line 578, in sample_dpmpp_sde
comfyui-rocm  |     x_2 = x_2 + noise_sampler(sigma_fn(t), sigma_fn(s)) * s_noise * su
comfyui-rocm  |   File "/root/ComfyUI/comfy/k_diffusion/sampling.py", line 124, in __call__
comfyui-rocm  |     return self.tree(t0, t1) / (t1 - t0).abs().sqrt()
comfyui-rocm  |   File "/root/ComfyUI/comfy/k_diffusion/sampling.py", line 95, in __call__
comfyui-rocm  |     w = torch.stack([tree(t0.cpu().float(), t1.cpu().float()).to(t0.dtype).to(t0.device) for tree in self.trees]) * (self.sign * sign)
comfyui-rocm  |   File "/root/ComfyUI/comfy/k_diffusion/sampling.py", line 95, in <listcomp>
comfyui-rocm  |     w = torch.stack([tree(t0.cpu().float(), t1.cpu().float()).to(t0.dtype).to(t0.device) for tree in self.trees]) * (self.sign * sign)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/derived.py", line 169, in __call__
comfyui-rocm  |     out = self._interval(t, tb, return_U=return_U, return_A=return_A)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 638, in __call__
comfyui-rocm  |     intervals = self._last_interval._loc(ta, tb)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 275, in _loc
comfyui-rocm  |     trampoline.trampoline(self._loc_inner(ta, tb, out))
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/trampoline/__init__.py", line 40, in trampoline
comfyui-rocm  |     raise exception
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/trampoline/__init__.py", line 26, in trampoline
comfyui-rocm  |     res = next(stack[-1])
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 296, in _loc_inner
comfyui-rocm  |     self._split(tb)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 328, in _split
comfyui-rocm  |     self._left_child._split(midway)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 328, in _split
comfyui-rocm  |     self._left_child._split(midway)
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 328, in _split
comfyui-rocm  |     self._left_child._split(midway)
comfyui-rocm  |   [Previous line repeated 953 more times]
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 323, in _split
comfyui-rocm  |     self._split_exact(0.5 * (self._end + self._start))
comfyui-rocm  |   File "/usr/lib/python3.10/site-packages/torchsde/_brownian/brownian_interval.py", line 336, in _split_exact
comfyui-rocm  |     generator = np.random.SeedSequence(entropy=self._top._entropy,
comfyui-rocm  |   File "bit_generator.pyx", line 315, in numpy.random.bit_generator.SeedSequence.__init__
comfyui-rocm  |   File "bit_generator.pyx", line 390, in numpy.random.bit_generator.SeedSequence.get_assembled_entropy
comfyui-rocm  |   File "bit_generator.pyx", line 148, in numpy.random.bit_generator._coerce_to_uint32_array
comfyui-rocm  |   File "bit_generator.pyx", line 140, in numpy.random.bit_generator._coerce_to_uint32_array
comfyui-rocm  |   File "bit_generator.pyx", line 79, in numpy.random.bit_generator._int_to_uint32_array
comfyui-rocm  | RecursionError: maximum recursion depth exceeded while calling a Python object
comfyui-rocm  | 
comfyui-rocm  | Prompt executed in 14.12 seconds