search

NVIDIA / Stable-Diffusion-WebUI-TensorRT

TensorRT Extension for Stable Diffusion Web UI
MIT License
1.91k stars 146 forks source link

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm) #197

Open Adzido opened 11 months ago

Adzido commented 11 months ago 
Traceback (most recent call last):
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\call_queue.py", line 57, in f
    res = list(func(*args, **kwargs))
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\call_queue.py", line 36, in f
    res = func(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\txt2img.py", line 55, in txt2img
    processed = processing.process_images(p)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\processing.py", line 732, in process_images
    res = process_images_inner(p)
  File "C:\Users\AdziOo\stable-diffusion-webui\extensions\sd-webui-controlnet\scripts\batch_hijack.py", line 42, in processing_process_images_hijack
    return getattr(processing, '__controlnet_original_process_images_inner')(p, *args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\processing.py", line 867, in process_images_inner
    samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
  File "C:\Users\AdziOo\stable-diffusion-webui\extensions\sd-webui-bmab\sd_bmab\sd_override\txt2img.py", line 38, in sample
    samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in sample
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_samplers_common.py", line 261, in launch_sampling
    return func()
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_samplers_kdiffusion.py", line 235, in <lambda>
    samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\utils\_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\sampling.py", line 594, in sample_dpmpp_2m
    denoised = model(x, sigmas[i] * s_in, **extra_args)
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_samplers_cfg_denoiser.py", line 169, in forward
    x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 112, in forward
    eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\repositories\k-diffusion\k_diffusion\external.py", line 138, in get_eps
    return self.inner_model.apply_model(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_models_xl.py", line 37, in apply_model
    return self.model(x, t, cond)
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Users\AdziOo\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\wrappers.py", line 28, in forward
    return self.diffusion_model(
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1538, in _call_impl
    result = forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\repositories\generative-models\sgm\modules\diffusionmodules\openaimodel.py", line 984, in forward
    emb = self.time_embed(t_emb)
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\container.py", line 217, in forward
    input = module(input)
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\AdziOo\stable-diffusion-webui\extensions-builtin\Lora\networks.py", line 429, in network_Linear_forward
    return originals.Linear_forward(self, input)
  File "C:\Users\AdziOo\stable-diffusion-webui\extensions\stable-diffusion-webui-composable-lora\composable_lora.py", line 150, in lora_Linear_forward
    return lora_forward(self, input, torch.nn.Linear_forward_before_lora(self, input))
  File "C:\Users\AdziOo\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)

I happened earlier to use TensorRT for SD 1.5 and SDXL now unfortunately such an error. Could the reason be that I made the engine for too high resolution? The model I am using should be used at 768x1280. Tryed reinstall and --device-id 0 in .bat. Still error. Any idea? My GPU is 4080.

arnolde commented 11 months ago

Same problem, I'm running Ubuntu 22.04 with RTX4060ti16GB and Nvidia driver 535.129.03

(base) ea@cm:/data/ai/stable-diffusion-webui$ ./webui.sh --listen --xformers --enable-insecure-extension-access

################################################################
Install script for stable-diffusion + Web UI
Tested on Debian 11 (Bullseye)
################################################################

################################################################
Running on ea user
################################################################

################################################################
Repo already cloned, using it as install directory
################################################################

################################################################
Create and activate python venv
################################################################

################################################################
Launching launch.py...
################################################################
Using TCMalloc: libtcmalloc_minimal.so.4
Python 3.11.5 (main, Sep 11 2023, 13:54:46) [GCC 11.2.0]
Version: v1.6.1
Commit hash: 4afaaf8a020c1df457bcf7250cb1c7f609699fa7
Launching Web UI with arguments: --listen --xformers --enable-insecure-extension-access
Loading weights [31e35c80fc] from /data/ai/stable-diffusion-webui/models/Stable-diffusion/sd_xl_base_1.0.safetensors
Creating model from config: /data/ai/stable-diffusion-webui/repositories/generative-models/configs/inference/sd_xl_base.yaml
Applying attention optimization: xformers... done.
Model loaded in 53.6s (load weights from disk: 10.1s, create model: 0.8s, apply weights to model: 41.3s, load textual inversion embeddings: 0.1s, calculate empty prompt: 1.1s).
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 85.9s (prepare environment: 11.4s, import torch: 6.1s, import gradio: 2.2s, setup paths: 4.5s, initialize shared: 0.2s, other imports: 3.1s, setup codeformer: 0.2s, list SD models: 0.2s, load scripts: 4.0s, initialize extra networks: 0.2s, create ui: 53.6s, gradio launch: 0.5s).
Activating unet: [TRT] sd_xl_base_1.0
Loading TensorRT engine: /data/ai/stable-diffusion-webui/models/Unet-trt/sd_xl_base_1.0_be9edd61_cc89_sample=1x4x96x96+2x4x128x128+8x4x128x128-timesteps=1+2+8-encoder_hidden_states=1x77x2048+2x77x2048+8x154x2048-y=1x2816+2x2816+8x2816.trt
[W] 'colored' module is not installed, will not use colors when logging. To enable colors, please install the 'colored' module: python3 -m pip install colored
[I] Loading bytes from /data/ai/stable-diffusion-webui/models/Unet-trt/sd_xl_base_1.0_be9edd61_cc89_sample=1x4x96x96+2x4x128x128+8x4x128x128-timesteps=1+2+8-encoder_hidden_states=1x77x2048+2x77x2048+8x154x2048-y=1x2816+2x2816+8x2816.trt
Profile 0:
    sample = [(1, 4, 96, 96), (2, 4, 128, 128), (8, 4, 128, 128)]
    timesteps = [(1,), (2,), (8,)]
    encoder_hidden_states = [(1, 77, 2048), (2, 77, 2048), (8, 154, 2048)]
    y = [(1, 2816), (2, 2816), (8, 2816)]
    latent = [(115), (115), (0)]

  0%|                                                                                                                          | 0/20 [00:00<?, ?it/s]
*** Error completing request
*** Arguments: ('task(0rejajwt1wtyf18)', 'a cat', '', [], 20, 'DPM++ 2M Karras', 1, 1, 7, 512, 512, False, 0.7, 2, 'Latent', 0, 0, 0, 'Use same checkpoint', 'Use same sampler', '', '', [], <gradio.routes.Request object at 0x7f4c9d41fcd0>, 0, False, '', 0.8, -1, False, -1, 0, 0, 0, False, False, 'positive', 'comma', 0, False, False, '', 1, '', [], 0, '', [], 0, '', [], True, False, False, False, 0, False) {}
    Traceback (most recent call last):
      File "/data/ai/stable-diffusion-webui/modules/call_queue.py", line 57, in f
        res = list(func(*args, **kwargs))
                   ^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/call_queue.py", line 36, in f
        res = func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/txt2img.py", line 55, in txt2img
        processed = processing.process_images(p)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/processing.py", line 732, in process_images
        res = process_images_inner(p)
              ^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/processing.py", line 867, in process_images_inner
        samples_ddim = p.sample(conditioning=p.c, unconditional_conditioning=p.uc, seeds=p.seeds, subseeds=p.subseeds, subseed_strength=p.subseed_strength, prompts=p.prompts)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/processing.py", line 1140, in sample
        samples = self.sampler.sample(self, x, conditioning, unconditional_conditioning, image_conditioning=self.txt2img_image_conditioning(x))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 235, in sample
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_samplers_common.py", line 261, in launch_sampling
        return func()
               ^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_samplers_kdiffusion.py", line 235, in <lambda>
        samples = self.launch_sampling(steps, lambda: self.func(self.model_wrap_cfg, x, extra_args=self.sampler_extra_args, disable=False, callback=self.callback_state, **extra_params_kwargs))
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/sampling.py", line 594, in sample_dpmpp_2m
        denoised = model(x, sigmas[i] * s_in, **extra_args)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_samplers_cfg_denoiser.py", line 169, in forward
        x_out = self.inner_model(x_in, sigma_in, cond=make_condition_dict(cond_in, image_cond_in))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 112, in forward
        eps = self.get_eps(input * c_in, self.sigma_to_t(sigma), **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/repositories/k-diffusion/k_diffusion/external.py", line 138, in get_eps
        return self.inner_model.apply_model(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_models_xl.py", line 37, in apply_model
        return self.model(x, t, cond)
               ^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_hijack_utils.py", line 17, in <lambda>
        setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
                                                                     ^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/modules/sd_hijack_utils.py", line 28, in __call__
        return self.__orig_func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/repositories/generative-models/sgm/modules/diffusionmodules/wrappers.py", line 28, in forward
        return self.diffusion_model(
               ^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/repositories/generative-models/sgm/modules/diffusionmodules/openaimodel.py", line 984, in forward
        emb = self.time_embed(t_emb)
              ^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/container.py", line 217, in forward
        input = module(input)
                ^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
        return forward_call(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 429, in network_Linear_forward
        return originals.Linear_forward(self, input)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/data/ai/stable-diffusion-webui/venv/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114, in forward
        return F.linear(input, self.weight, self.bias)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm)
---

EDIT: The above attempt was using 512x512 with a TRT model built for 768-1024, but another attempt with a 1024x1024 prompt gave the same result. NOTE: During installation: a) I did not generate the "default" TRT model but a different selection (768-1024, batch 1-4) - not sure if that matters. b) During generation, after more than 5mins my PC stopped doing any work for more than a minute (no process using notable CPU) so I switched to the main webui page, hope that didnt break anything

arnolde commented 11 months ago

I just realized this is a duplicate of https://github.com/NVIDIA/Stable-Diffusion-WebUI-TensorRT/issues/97 Short recap: SDXL is only supported if you use the dev branch of A1111. The suggested solution is to do a "git fetch dev", "git switch dev", delete the venv folder and restart the webui. I hesitate because I don't like re-downloading GB's of stuff and recompiling everything just for such a minor change, I wonder if there is a more economic alternative?

Adzido commented 11 months ago

I just realized this is a duplicate of #97 Short recap: SDXL is only supported if you use the dev branch of A1111. The suggested solution is to do a "git fetch dev", "git switch dev", delete the venv folder and restart the webui. I hesitate because I don't like re-downloading GB's of stuff and recompiling everything just for such a minor change, I wonder if there is a more economic alternative?

I thought that those posts that talk about changing the version of A1111 are some very old ones. So still to use SDXL TRT you have to use that? That sounds weak. Especially since half of the other plugins will probably stop giving by not being adapted to this version of A1111.

arnolde commented 11 months ago

It says right on the project's README on the first page: "Native SDXL support coming in a future release. Please use the dev branch if you would like to use it today."

jason571 commented 10 months ago

Same problem: Dev branch return self._call_impl(*args, kwargs) File "/home/flyang/.local/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl return forward_call(*args, *kwargs) File "/mnt/p/home/flyang/stable-diffusion-webui/modules/sd_hijack_utils.py", line 17, in setattr(resolved_obj, func_path[-1], lambda args, kwargs: self(*args, *kwargs)) File "/mnt/p/home/flyang/stable-diffusion-webui/modules/sd_hijack_utils.py", line 28, in call return self.__orig_func(args, **kwargs) File "/home/flyang/.local/lib/python3.10/site-packages/ldm/modules/diffusionmodules/util.py", line 219, in forward return super().forward(x.float()).type(x.dtype) File "/mnt/p/home/flyang/stable-diffusion-webui/extensions-builtin/Lora/networks.py", line 459, in network_GroupNorm_forward return originals.GroupNorm_forward(self, input) File "/home/flyang/.local/lib/python3.10/site-packages/torch/nn/modules/normalization.py", line 279, in forward return F.group_norm( File "/home/flyang/.local/lib/python3.10/site-packages/torch/nn/functional.py", line 2558, in group_norm return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled) RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper_CUDA__native_group_norm)

FMsunyh commented 10 months ago

Same problem

Dev branch

webui-docker-auto-1  | ============= Diagnostic Run torch.onnx.export version 2.0.1+cu118 =============                                                                                    
webui-docker-auto-1  | verbose: False, log level: Level.ERROR                                                                                                                              
webui-docker-auto-1  | ======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================                                                                                    
webui-docker-auto-1  |                                                                                                                                                                     
webui-docker-auto-1  | ERROR:root:Exporting to ONNX failed. Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argu
ment mat1 in method wrapper_CUDA_addmm)                                                                                                                                                    
webui-docker-auto-1  | Building TensorRT engine... This can take a while, please check the progress in the terminal.                                                                       
webui-docker-auto-1  | Building TensorRT engine for /stable-diffusion-webui/models/Unet-onnx/v1-5-pruned-emaonly.onnx: /stable-diffusion-webui/models/Unet-trt/v1-5-pruned-emaonly_d7049739_cc75_sample=2x4x64x64-timesteps=2-encoder_hidden_states=2x77x768.trt
webui-docker-auto-1  | Could not open file /stable-diffusion-webui/models/Unet-onnx/v1-5-pruned-emaonly.onnx
webui-docker-auto-1  | Could not open file /stable-diffusion-webui/models/Unet-onnx/v1-5-pruned-emaonly.onnx
webui-docker-auto-1  | [E] ModelImporter.cpp:773: Failed to parse ONNX model from file: /stable-diffusion-webui/models/Unet-onnx/v1-5-pruned-emaonly.onnx
webui-docker-auto-1  | [!] Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model?
webui-docker-auto-1  | Traceback (most recent call last):
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/routes.py", line 488, in run_predict
webui-docker-auto-1  |     output = await app.get_blocks().process_api(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1431, in process_api
webui-docker-auto-1  |     result = await self.call_function(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/blocks.py", line 1103, in call_function
webui-docker-auto-1  |     prediction = await anyio.to_thread.run_sync(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/to_thread.py", line 33, in run_sync
webui-docker-auto-1  |     return await get_asynclib().run_sync_in_worker_thread(
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 877, in run_sync_in_worker_thread
webui-docker-auto-1  |     return await future
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 807, in run
webui-docker-auto-1  |     result = context.run(func, *args)
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/gradio/utils.py", line 707, in wrapper
webui-docker-auto-1  |     response = f(*args, **kwargs)
webui-docker-auto-1  |   File "/stable-diffusion-webui/extensions/Stable-Diffusion-WebUI-TensorRT/ui_trt.py", line 126, in export_unet_to_trt
webui-docker-auto-1  |     ret = export_trt(
webui-docker-auto-1  |   File "/stable-diffusion-webui/extensions/Stable-Diffusion-WebUI-TensorRT/exporter.py", line 231, in export_trt
webui-docker-auto-1  |     ret = engine.build(
webui-docker-auto-1  |   File "/stable-diffusion-webui/extensions/Stable-Diffusion-WebUI-TensorRT/utilities.py", line 227, in build
webui-docker-auto-1  |     network = network_from_onnx_path(
webui-docker-auto-1  |   File "<string>", line 3, in network_from_onnx_path
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/polygraphy/backend/base/loader.py", line 40, in __call__
webui-docker-auto-1  |     return self.call_impl(*args, **kwargs)
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/polygraphy/util/util.py", line 710, in wrapped
webui-docker-auto-1  |     return func(*args, **kwargs)
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/polygraphy/backend/trt/loader.py", line 227, in call_impl
webui-docker-auto-1  |     trt_util.check_onnx_parser_errors(parser, success)
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/polygraphy/backend/trt/util.py", line 86, in check_onnx_parser_errors
webui-docker-auto-1  |     G_LOGGER.critical("Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model?")
webui-docker-auto-1  |   File "/usr/local/lib/python3.10/site-packages/polygraphy/logger/logger.py", line 605, in critical
webui-docker-auto-1  |     raise ExceptionType(message) from None
webui-docker-auto-1  | polygraphy.exception.exception.PolygraphyException: Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model?
Thetail001 commented 10 months ago

+1,when try to generate SDXL

ERROR:root:Exporting to ONNX failed. Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_CUDA_addmm) Building TensorRT engine... This can take a while, please check the progress in the terminal. Building TensorRT engine for D:\stable-diffusion-webui\models\Unet-onnx\sd_xl_base_1.0.onnx: D:\stable-diffusion-webui\models\Unet-trt\sd_xl_base_1.0_be9edd61_cc89_sample=2x4x128x128+2x4x128x128+2x4x128x128-timesteps=2+2+2-encoder_hidden_states=2x77x2048+2x77x2048+2x77x2048-y=2x2816+2x2816+2x2816.trt Could not open file D:\stable-diffusion-webui\models\Unet-onnx\sd_xl_base_1.0.onnx Could not open file D:\stable-diffusion-webui\models\Unet-onnx\sd_xl_base_1.0.onnx [E] ModelImporter.cpp:773: Failed to parse ONNX model from file: D:\stable-diffusion-webui\models\Unet-onnx\sd_xl_base_1.0.onnx [!] Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model? Traceback (most recent call last): File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\routes.py", line 488, in run_predict output = await app.get_blocks().process_api( File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1431, in process_api result = await self.call_function( File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\blocks.py", line 1103, in call_function prediction = await anyio.to_thread.run_sync( File "D:\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 33, in run_sync return await get_asynclib().run_sync_in_worker_thread( File "D:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 877, in run_sync_in_worker_thread return await future File "D:\stable-diffusion-webui\venv\lib\site-packages\anyio_backends_asyncio.py", line 807, in run result = context.run(func, args) File "D:\stable-diffusion-webui\venv\lib\site-packages\gradio\utils.py", line 707, in wrapper response = f(args, kwargs) File "D:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\ui_trt.py", line 126, in export_unet_to_trt ret = export_trt( File "D:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\exporter.py", line 231, in export_trt ret = engine.build( File "D:\stable-diffusion-webui\extensions\Stable-Diffusion-WebUI-TensorRT\utilities.py", line 227, in build network = network_from_onnx_path( File "", line 3, in network_from_onnx_path File "D:\stable-diffusion-webui\venv\lib\site-packages\polygraphy\backend\base\loader.py", line 40, in call return self.call_impl(*args, *kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\polygraphy\util\util.py", line 710, in wrapped return func(args, kwargs) File "D:\stable-diffusion-webui\venv\lib\site-packages\polygraphy\backend\trt\loader.py", line 227, in call_impl trt_util.check_onnx_parser_errors(parser, success) File "D:\stable-diffusion-webui\venv\lib\site-packages\polygraphy\backend\trt\util.py", line 86, in check_onnx_parser_errors G_LOGGER.critical("Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model?") File "D:\stable-diffusion-webui\venv\lib\site-packages\polygraphy\logger\logger.py", line 605, in critical raise ExceptionType(message) from None polygraphy.exception.exception.PolygraphyException: Failed to parse ONNX model. Does the model file exist and contain a valid ONNX model?

hahn23 commented 10 months ago

Hi,

i faced the same issue today but I think I found an workaround for the moment. You just have to use the DEV branch of this WebUi extension. Thats how I done it:

Open an cmd and "cd" to the TensorRT extension folder. Than do a "git checkout -f dev" and to be on the safe side, add an "git pull" afterwards. (during those steps the webui is off!). Just start Webui again and it should work. At least it did for me. With webui master branch and dev branch of the extension.

Jonseed commented 10 months ago

I'm also getting this error. I tried the DEV branch of the extension, and that didn't work either.

449693691 commented 9 months ago

I'm also getting this error. I tried the DEV branch of the extension, and that didn't work either. Me too

bigmover commented 9 months ago

Anyone can solve it?

449693691 commented 9 months ago

Anyone can solve it?

I solved the problem by turning off video memory optimization.

bigmover commented 9 months ago

Anyone can solve it?

I solved the problem by turning off video memory optimization.

Are you on the mode of 'webui' or api?

Kaiten10 commented 6 months ago

Using --lowvram fixed it for me. Since the error message says it found two devices (cpu and cuda:0), I think --lowvram fixed it by forcing SD to only use the CPU.

sup2069 commented 4 months ago

Removing the following conflicting arguments from my webui-user.sh allowed me to export the engine!

 --no-half --medvram --opt-split-attention