ArtVentureX / sd-webui-agent-scheduler

629 stars 66 forks source link

running failed with exception Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm), Traceback (most recent call last) #138

Open 296651310 opened 1 year ago

296651310 commented 1 year ago

The API call encountered an error. sd-webui version 1.3.2

artventuredev commented 1 year ago

Maybe related to this https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/11162. Can you post your boot logs?

296651310 commented 1 year ago

[2023-09-21 13:51:57] 2023-09-21 05:51:57,316 - sdwebui - INFO - saving image data-oss/outputs/img2img-images/2023-09-21/055156-af9b7804-d326-4d97-9c39-85b465618658-3341537683 time 0.6s (_atomically_save_image: 0.5s) [2023-09-21 13:51:57] load network JanPawel2v21 from data-oss/models/Lora/JanPawel2v21.safetensors with 0.6 0.6 None [2023-09-21 13:51:58] 0%| | 0/16 [00:00<?, ?it/s] [2023-09-21 13:52:01] Total progress: 97it [47:33, 2.51s/it] 6%|▋ | 1/16 [00:02<00:38, 2.57s/it] [2023-09-21 13:52:03] Total progress: 98it [47:36, 2.53s/it] 12%|█▎ | 2/16 [00:05<00:35, 2.57s/it] [2023-09-21 13:52:06] Total progress: 99it [47:38, 2.54s/it] 19%|█▉ | 3/16 [00:07<00:33, 2.56s/it] [2023-09-21 13:52:08] Total progress: 100it [47:41, 2.55s/it] 25%|██▌ | 4/16 [00:10<00:30, 2.57s/it] [2023-09-21 13:52:11] Total progress: 101it [47:44, 2.55s/it] 31%|███▏ | 5/16 [00:12<00:28, 2.57s/it] [2023-09-21 13:52:14] Total progress: 102it [47:46, 2.56s/it] 38%|███▊ | 6/16 [00:15<00:25, 2.57s/it] [2023-09-21 13:52:16] Total progress: 103it [47:49, 2.56s/it]2023-09-21 05:52:16,320 - Inpaint Anything - INFO - input_image: (768, 512, 3) uint8 [2023-09-21 13:52:16] 38%|███▊ | 6/16 [00:19<00:31, 3.17s/it] [2023-09-21 13:52:16] 2023-09-21 05:52:16,966 - sdwebui - INFO - Ending job Batch 3 out of 4 (105.91 seconds) [2023-09-21 13:52:17] 2023-09-21 05:52:17,166 - sdwebui - ERROR - running failed with exception Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm), Traceback (most recent call last): [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/api/api.py", line 827, in img2imgapi [2023-09-21 13:52:17] processed = process_images(p) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/processing.py", line 677, in process_images [2023-09-21 13:52:17] res = process_images_inner(p) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/extensions-builtin/sd-webui-controlnet/scripts/batch_hijack.py", line 42, in processing_process_images_hijack [2023-09-21 13:52:17] return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/processing.py", line 797, in process_images_inner [2023-09-21 13:52:17] samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/extensions-builtin/sd-webui-controlnet/scripts/hook.py", line 350, in process_sample [2023-09-21 13:52:17] return process.sample_before_CN_hack(*args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/processing.py", line 1374, in sample [2023-09-21 13:52:17] samples = self.sampler.sample_img2img(self, self.init_latent, x, conditioning, unconditional_conditioning, image_conditioning=self.image_conditioning) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 434, in sample_img2img [2023-09-21 13:52:17] samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, extra_params_kwargs)) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 303, in launch_sampling [2023-09-21 13:52:17] return func() [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 434, in [2023-09-21 13:52:17] samples = self.launch_sampling(t_enc + 1, lambda: self.func(self.model_wrap_cfg, xi, extra_args=extra_args, disable=False, callback=self.callback_state, *extra_params_kwargs)) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 115, in decorate_context [2023-09-21 13:52:17] return func(args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/sampling.py", line 534, in sample_dpmpp_2s_ancestral [2023-09-21 13:52:17] denoised_2 = model(x_2, sigma_fn(s) * s_in, *extra_args) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(args, **kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 205, in forward

[2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(*args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 112, in forward [2023-09-21 13:52:17] eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), *kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 138, in get_eps [2023-09-21 13:52:17] return self.inner_model.apply_model(args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_hijack_utils.py", line 17, in [2023-09-21 13:52:17] setattr(resolved_obj, func_path[-1], lambda *args, kwargs: self(*args, *kwargs)) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/modules/sd_hijack_utils.py", line 28, in call [2023-09-21 13:52:17] return self.__orig_func(args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 858, in apply_model [2023-09-21 13:52:17] x_recon = self.model(x_noisy, t, cond) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(*args, *kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/repositories/stable-diffusion-stability-ai/ldm/models/diffusion/ddpm.py", line 1335, in forward [2023-09-21 13:52:17] out = self.diffusion_model(x, t, context=cc) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/extensions-builtin/sd-webui-controlnet/scripts/hook.py", line 661, in forward_webui [2023-09-21 13:52:17] return forward(*args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/extensions-builtin/sd-webui-controlnet/scripts/hook.py", line 577, in forward [2023-09-21 13:52:17] emb = self.time_embed(t_emb) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(*args, *kwargs) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/container.py", line 217, in forward [2023-09-21 13:52:17] input = module(input) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1501, in _call_impl [2023-09-21 13:52:17] return forward_call(args, kwargs) [2023-09-21 13:52:17] File "/code/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 390, in network_Linear_forward [2023-09-21 13:52:17] return torch.nn.Linear_forward_before_network(self, input) [2023-09-21 13:52:17] File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/linear.py", line 114, in forward [2023-09-21 13:52:17] return F.linear(input, self.weight, self.bias) [2023-09-21 13:52:17] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm) [2023-09-21 13:52:17] , vram free: 17794.125M, vram total: 22731.875M [2023-09-21 13:52:17] 2023-09-21 05:52:17,166 - sdwebui - INFO - method:image2image, width:750, height:1000, batchsize:1, n_iter:4, id_task:None, uid:None, sd_vae:None: 106.1s (model inference end: 106.0s), hostname: sdnew-75564868ff-fvjld [2023-09-21 13:52:17] ERROR - [AgentScheduler] Task b81f238c-910c-4383-9fd3-a4cef0e134e3 failed:

296651310 commented 1 year ago

The error log looks something like this. I would greatly appreciate it if you could help clarify my doubts

artventuredev commented 1 year ago

I've noticed that others have experienced similar issues with version 1.3.2: https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/11080.

I recommend updating the webui or installing it in a new temporary folder to check if this resolves the problem.