vladmandic
commented
1 year ago This error comes from inside Controlnet - do you have it enabled? Is there an issue if controlnet is NOT enabled?
Closed ledrose closed 1 year ago
vladmandic
commented
1 year ago This error comes from inside Controlnet - do you have it enabled? Is there an issue if controlnet is NOT enabled?
ledrose
commented
1 year ago If ControlNet is disabled then error jumps into Lora extension. If all extensions are disabled then the error is still there. Log without any extensions:
21:35:48-503437 ERROR Exception: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
21:35:48-504665 ERROR Arguments: args=('task(273ex59ce7qmkzt)', '1', '', [], 20, None, None, True, False, False, 1, 1, 6, 6, 0.7, 1, -1.0, -1.0, 0, 0, 0, 512, 512, False, 0.3, 2, 'None', 20, 0, 0, 5, 0.8, '', '', [], 0, False, False,
'positive', 'comma', 0, False, False, '', 0, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) kwargs={}
21:35:48-506845 ERROR gradio call: RuntimeError
╭───────────────────────────────────────────────────────────────────────────────── Traceback (most recent call last) ──────────────────────────────────────────────────────────────────────────────────╮
│ /home/regedit/StabilityMatrix/Packages/automatic/modules/call_queue.py:34 in f │
│ │
│ 33 │ │ │ try: │
│ ❱ 34 │ │ │ │ res = func(*args, **kwargs) │
│ 35 │ │ │ │ progress.record_results(id_task, res) │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/modules/txt2img.py:65 in txt2img │
│ │
│ 64 │ if processed is None: │
│ ❱ 65 │ │ processed = processing.process_images(p) │
│ 66 │ p.close() │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/modules/processing.py:622 in process_images │
│ │
│ 621 │ │ else: │
│ ❱ 622 │ │ │ res = process_images_inner(p) │
│ 623 │ finally: │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/modules/processing.py:759 in process_images_inner │
│ │
│ 758 │ │ │ │ with devices.without_autocast() if devices.unet_needs_upcast else device │
│ ❱ 759 │ │ │ │ │ samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u │
│ 760 │ │ │ │ x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to( │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/modules/processing.py:994 in sample │
│ │
│ 993 │ │ x = create_random_tensors([4, self.height // 8, self.width // 8], seeds=seeds, s │
│ ❱ 994 │ │ samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, │
│ 995 │ │ if not self.enable_hr or shared.state.interrupted or shared.state.skipped: │
│ │
│ ... 19 frames hidden ... │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/repositories/stable-diffusion-stability-ai/ldm/modules/diffusionmodules/openaimodel.py:789 in forward │
│ │
│ 788 │ │ t_emb = timestep_embedding(timesteps, self.model_channels, repeat_only=False) │
│ ❱ 789 │ │ emb = self.time_embed(t_emb) │
│ 790 │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/container.py:217 in forward │
│ │
│ 216 │ │ for module in self: │
│ ❱ 217 │ │ │ input = module(input) │
│ 218 │ │ return input │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/module.py:1501 in _call_impl │
│ │
│ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): │
│ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) │
│ 1502 │ │ # Do not call functions when jit is used │
│ │
│ /home/regedit/StabilityMatrix/Packages/automatic/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py:114 in forward │
│ │
│ 113 │ def forward(self, input: Tensor) -> Tensor: │
│ ❱ 114 │ │ return F.linear(input, self.weight, self.bias) │
│ 115 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
I tried different commits and it seams that this problem first materialized in commit f36c1eb4762e7d63f0bc39ede315c576070e1038 "jumbo patch"
ledrose
commented
1 year ago It also seems to be a problem only with UniPC sampler. If you switch models while UniPC selected this error appears. But if you then switch to another sampler and generate a picture and then switch sampler back to UniPC the error goes away.
jasl
commented
1 year ago Same issue here. I just pulled the latest commit and then got the issue.
Resolved by git checkout 5b41115bc89ed6e730aa18d8445d4bd2a9925b15 (revert to https://github.com/vladmandic/automatic/commit/5b41115bc89ed6e730aa18d8445d4bd2a9925b15)
I'm using Ubuntu server 22.04, ./webui.sh --listen --backend=original --safe
mart-hill
commented
1 year ago e56dd554 commit: I noticed that I have to switch the sampler (from 'Euler a' in my case, because I "restored" the parameters) to 'UniPC' or anything else than 'Euler a', and generate an image after EVERY model change, otherwise I'll get the error shown up there. Sometimes I also get an error (in addition to the one above):
21:54:28-625799 ERROR API error: GET:
http://localhost:7860/infinite_image_browsing/image-thumbnail?path=X%3A%5CAI%5Cstable-diffusion-ui%5Coutputs%5Ctxt2
img-images%5C2023-09-08%5C00037-20230908215427_926267763_53afb136.webp&size=512x512&t=2023-09-08%2021%3A54%3A26
{'error': 'UnidentifiedImageError', 'detail': '', 'body': '', 'errors': "cannot identify image file
'X:\\\\AI\\\\stable-diffusion-ui\\\\outputs\\\\txt2img-images\\\\2023-09-08\\\\00037-20230908215427_926267763_53afb
136.webp'"}
21:54:28-630300 ERROR HTTP API: UnidentifiedImageError
╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ X:\AI\automatic\venv\lib\site-packages\starlette\middleware\base.py:78 in call_next │
│ │
│ X:\AI\automatic\venv\lib\site-packages\anyio\streams\memory.py:98 in receive │
│ │
│ X:\AI\automatic\venv\lib\site-packages\anyio\streams\memory.py:91 in receive_nowait │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
EndOfStream
During handling of the above exception, another exception occurred:
╭─────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮
│ X:\AI\automatic\modules\middleware.py:41 in log_and_time │
│ │
│ 40 │ │ │ ts = time.time() │
│ ❱ 41 │ │ │ res: Response = await call_next(req) │
│ 42 │ │ │ duration = str(round(time.time() - ts, 4)) │
│ │
│ X:\AI\automatic\venv\lib\site-packages\starlette\middleware\base.py:84 in call_next │
│ │
│ X:\AI\automatic\venv\lib\site-packages\starlette\middleware\base.py:70 in coro │
│ │
│ X:\AI\automatic\venv\lib\site-packages\starlette\middleware\cors.py:83 in __call__ │
│ │
│ X:\AI\automatic\venv\lib\site-packages\starlette\middleware\gzip.py:24 in __call__ │
│ │
│ ... 7 frames hidden ... │
│ │
│ X:\AI\automatic\venv\lib\site-packages\starlette\routing.py:66 in app │
│ │
│ X:\AI\automatic\venv\lib\site-packages\fastapi\routing.py:273 in app │
│ │
│ X:\AI\automatic\venv\lib\site-packages\fastapi\routing.py:190 in run_endpoint_function │
│ │
│ X:\AI\automatic\extensions\sd-webui-infinite-image-browsing\scripts\iib\api.py:428 in thumbnail │
│ │
│ 427 │ │ # 如果缓存文件不存在，则生成缩略图并保存 │
│ ❱ 428 │ │ with Image.open(path) as img: │
│ 429 │ │ │ w, h = size.split("x") │
│ │
│ X:\AI\automatic\venv\lib\site-packages\PIL\Image.py:3298 in open │
│ │
│ 3297 │ msg = "cannot identify image file %r" % (filename if filename else fp) │
│ ❱ 3298 │ raise UnidentifiedImageError(msg) │
│ 3299 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
UnidentifiedImageError: cannot identify image file
'X:\\AI\\stable-diffusion-ui\\outputs\\txt2img-images\\2023-09-08\\00037-20230908215427_926267763_53afb136.webp'Image gets created though.
demoran23
commented
1 year ago I am receiving this error as well.
16:56:52-723664 ERROR Exception: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when
checking argument for argument mat1 in method wrapper_CUDA_addmm)
16:56:52-724663 ERROR Arguments: args=('task(kgqtoi1a6xhahtl)', 'girl', '', ['showcase'], 16, 4, None, True, False, False, 4, 4, 6, 6,
0.7, 1, -1.0, -1.0, 0, 0, 0, 768, 512, False, 0.3, 2.5, '4x-UltraSharp', 0, 0, 0, 5, 0.8, '', '', [], 0, False,
'Mixture of Diffusers', False, True, 1024, 1024, 128, 128, 32, 8, '4x-UltraSharp', 2, False, 10, 1, 1, 64, False,
False, False, False, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '',
'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '',
'', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2,
'', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2, 0.2, '', '', 'Background', 0.2, -1.0, False, 0.4, 0.4, 0.2,
0.2, '', '', 'Background', 0.2, -1.0, False, 800, 96, True, True, True, False,
<scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000026092C31330>,
<scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000026093705C90>,
<scripts.controlnet_ui.controlnet_ui_group.UiControlNetUnit object at 0x0000026093707D90>, False, {'ad_model':
'face_yolov8n.pt', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0,
'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4,
'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512,
'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7,
'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_sampler': False, 'ad_sampler': 'UniPC',
'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1,
'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious',
'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()},
{'ad_model': 'None', 'ad_prompt': '', 'ad_negative_prompt': '', 'ad_confidence': 0.3, 'ad_mask_k_largest': 0,
'ad_mask_min_ratio': 0, 'ad_mask_max_ratio': 1, 'ad_x_offset': 0, 'ad_y_offset': 0, 'ad_dilate_erode': 4,
'ad_mask_merge_invert': 'None', 'ad_mask_blur': 4, 'ad_denoising_strength': 0.4, 'ad_inpaint_only_masked': True,
'ad_inpaint_only_masked_padding': 32, 'ad_use_inpaint_width_height': False, 'ad_inpaint_width': 512,
'ad_inpaint_height': 512, 'ad_use_steps': False, 'ad_steps': 28, 'ad_use_cfg_scale': False, 'ad_cfg_scale': 7,
'ad_use_checkpoint': False, 'ad_checkpoint': 'Use same checkpoint', 'ad_use_sampler': False, 'ad_sampler': 'UniPC',
'ad_use_noise_multiplier': False, 'ad_noise_multiplier': 1, 'ad_use_clip_skip': False, 'ad_clip_skip': 1,
'ad_restore_face': False, 'ad_controlnet_model': 'None', 'ad_controlnet_module': 'inpaint_global_harmonious',
'ad_controlnet_weight': 1, 'ad_controlnet_guidance_start': 0, 'ad_controlnet_guidance_end': 1, 'is_api': ()}, True,
False, 1, False, False, False, 1.1, 1.5, 100, 0.7, False, True, True, False, False, 0,
'Gustavosta/MagicPrompt-Stable-Diffusion', '', 0, 0, 0, 0, 0.25, False, 0, False, False, 'positive', 'comma', 0,
False, False, '', 4, '04,08,12,14,16', [], 7, '-1, -1, -1', [], 0, '', [], True, False, False, False, 0, False,
None, None, False, None, None, False, None, None, False, 50, 5, 'all', 'all', 'all', '', '', '', '1', 'none', False, '', '', 'comma', '', True, '', '20', 'all', 'all', 'all', 'all', 0) kwargs={}
16:56:52-742663 ERROR gradio call: RuntimeError
╭──────────────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────────────╮│ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\modules\call_queue.py:34 in f ││ ││ 33 │ │ │ try: ││ ❱ 34 │ │ │ │ res = func(*args, **kwargs) ││ 35 │ │ │ │ progress.record_results(id_task, res) ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\modules\txt2img.py:65 in txt2img ││ ││ 64 │ if processed is None: ││ ❱ 65 │ │ processed = processing.process_images(p) ││ 66 │ p.close() ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\modules\processing.py:622 in process_images ││ ││ 621 │ │ else: ││ ❱ 622 │ │ │ res = process_images_inner(p) ││ 623 │ finally: ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\extensions-builtin\sd-webui-controlnet\scripts\batch_hijack.py:42 ││ in processing_process_images_hijack ││ ││ 41 │ │ │ # we are not in batch mode, fallback to original function ││ ❱ 42 │ │ │ return getattr(processing, '__controlnet_original_process_images_inner')(p, ││ 43 ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\modules\processing.py:759 in process_images_inner ││ ││ 758 │ │ │ │ with devices.without_autocast() if devices.unet_needs_upcast else device ││ ❱ 759 │ │ │ │ │ samples_ddim = p.sample(conditioning=c, unconditional_conditioning=u ││ 760 │ │ │ │ x_samples_ddim = [decode_first_stage(p.sd_model, samples_ddim[i:i+1].to( ││ ││ ... 19 frames hidden ... ││ ││ C:\Users\adams\.pyenv\pyenv-win\versions\3.10.6\lib\site-packages\torch\nn\modules\container.py:217 in forward ││ ││ 216 │ │ for module in self: ││ ❱ 217 │ │ │ input = module(input) ││ 218 │ │ return input ││ ││ C:\Users\adams\.pyenv\pyenv-win\versions\3.10.6\lib\site-packages\torch\nn\modules\module.py:1501 in _call_impl ││ ││ 1500 │ │ │ │ or _global_forward_hooks or _global_forward_pre_hooks): ││ ❱ 1501 │ │ │ return forward_call(*args, **kwargs) ││ 1502 │ │ # Do not call functions when jit is used ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\extensions-builtin\Lora\lora.py:403 in lora_Linear_forward ││ ││ 402 │ ││ ❱ 403 │ return torch.nn.Linear_forward_before_lora(self, input) ││ 404 ││ ││ C:\Users\adams\AppData\Roaming\StabilityMatrix\Packages\SD.Next Web UI\extensions-builtin\a1111-sd-webui-lycoris\lycoris.py:741 in ││ lyco_Linear_forward ││ ││ 740 │ ││ ❱ 741 │ return torch.nn.Linear_forward_before_lyco(self, input) ││ 742 ││ ││ C:\Users\adams\.pyenv\pyenv-win\versions\3.10.6\lib\site-packages\torch\nn\modules\linear.py:114 in forward ││ ││ 113 │ def forward(self, input: Tensor) -> Tensor: ││ ❱ 114 │ │ return F.linear(input, self.weight, self.bias) ││ 115 │╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for
argument mat1 in method wrapper_CUDA_addmm)I don't have controlnet or tiled diffusion or tiled vae enabled. I'm using DPM++ 2M Karras. It happens when queueing through agent scheduler as well.
commit e56dd5544db6983c11af0667298b80223f168e1b.
vladmandic
commented
1 year ago should be fixed now.
f1am3d
commented
7 months ago Not fixed
vladmandic
commented
7 months ago commenting on am issue closed 7months ago doesnt help - document and create new issue.
f1am3d
commented
7 months ago I'm sorry, what does the date have to do with the issue?
vladmandic
commented
7 months ago that issue can occure for large number of reasons, this specific issue is resolved. if you're running into same problem, root cause needs to be investigated - which means submit issue with full required details.
Issue Description
Hello. After one of recent updates image generation broke. After first model load during startup everything works normally but after next model switch in UI any generation throws an error: "Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)"
Version Platform Description
Quotes from sdnext.log:
Relevant log output
Backend
Original
Model
SD 1.5
Acknowledgements