[Bug]: xyz grid is broken when using checkpoints as a parameter

PestToast commented 1 year ago

Is there an existing issue for this?

[X] I have searched the existing issues and checked the recent builds/commits

What happened?

When trying to use checkpoints as an axis in a grid for any axis other that "X" (or in the "X" axis with any other axes active), an error message appears and the generation preview is halted. In the console, the generation continues and eventually it is output to the webui, but the previews are stopped, which is a problem if I am making a large grid and I might notice a problem part way through that would cause me to interrupt the generation.

Steps to reproduce the problem

Generate a xyz grid with multiple checkpoints along the Y axis

What should have happened?

No error message and image previews appearing in the webui

Commit where the problem happens

a9fed7c3

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Brave

Command Line Arguments

--no-half --xformers

List of extensions

openOutpaint-webUI-extension openpose-editor posex sd-dynamic-prompts sd-webui-controlnet sd-webui-depth-lib sd-webui-model-converter sd_dreambooth_extension stable-diffusion-webui-tokenizer ultimate-upscale-for-automatic1111

Console logs

venv "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Commit hash: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2
Installing requirements for Web UI

#######################################################################################################
Initializing Dreambooth
If submitting an issue on github, please provide the below text for debugging purposes:

Python revision: 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Dreambooth revision: bff61c6b92b79ece2e140ee240e0628dd9a4ebef
SD-WebUI revision: a9fed7c364061ae6efb37f797b6b522cb3cf7aa2

Checking Dreambooth requirements...
Ignoring tensorflow-macos: markers 'sys_platform == "darwin" and platform_machine == "arm64"' don't match your environment
Ignoring mediapipe-silicon: markers 'sys_platform == "darwin"' don't match your environment
Collecting accelerate==0.16.0
  Using cached accelerate-0.16.0-py3-none-any.whl (199 kB)
Collecting transformers~=4.26.1
  Using cached transformers-4.26.1-py3-none-any.whl (6.3 MB)
Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
Installing collected packages: requests, accelerate, transformers
  Attempting uninstall: requests
    Found existing installation: requests 2.25.1
    Uninstalling requests-2.25.1:
      Successfully uninstalled requests-2.25.1
  Attempting uninstall: accelerate
    Found existing installation: accelerate 0.12.0
    Uninstalling accelerate-0.12.0:
      Successfully uninstalled accelerate-0.12.0
  Attempting uninstall: transformers
    Found existing installation: transformers 4.25.1
    Uninstalling transformers-4.25.1:
      Successfully uninstalled transformers-4.25.1
Successfully installed accelerate-0.16.0 requests-2.28.2 transformers-4.26.1

[+] torch version 1.13.1+cu117 installed.
[+] torchvision version 0.14.1+cu117 installed.
[+] accelerate version 0.16.0 installed.
[+] bitsandbytes version 0.35.4 installed.
[+] diffusers version 0.13.1 installed.
[+] transformers version 4.26.1 installed.
[+] xformers version 0.0.17.dev464 installed.

#######################################################################################################

Launching Web UI with arguments: --no-half --xformers
Script path is C:\Users\user\Desktop\git\stable-diffusion-webui
Loading weights [d725be5d18] from C:\Users\user\Desktop\git\stable-diffusion-webui\models\Stable-diffusion\revAnimated_v11.safetensors
Creating model from config: C:\Users\user\Desktop\git\stable-diffusion-webui\configs\v1-inference.yaml
LatentDiffusion: Running in eps-prediction mode
DiffusionWrapper has 859.52 M params.
Loading VAE weights specified in settings: C:\Users\user\Desktop\git\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.ckpt
Applying xformers cross attention optimization.
Textual inversion embeddings loaded(5): bad_prompt, bad_prompt_version2
Model loaded in 6.3s (create model: 0.6s, apply weights to model: 1.7s, load VAE: 0.4s, move model to device: 2.2s, load textual inversion embeddings: 1.3s).
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Startup time: 31.1s (import gradio: 2.3s, import ldm: 3.7s, other imports: 2.0s, list extensions: 1.4s, setup codeformer: 0.1s, load scripts: 3.6s, load SD checkpoint: 6.7s, create ui: 11.1s, gradio launch: 0.1s).
X/Y/Z plot will create 2 images on 1 1x2 grid. (Total steps to process: 20)
                                  Loading weights [9aba26abdf] from C:\Users\user\Desktop\git\stable-diffusion-webui\models\Stable-diffusion\deliberate_v2.safetensorsgress: 0it [00:00, ?it/s]
Loading VAE weights specified in settings: C:\Users\user\Desktop\git\stable-diffusion-webui\models\VAE\vae-ft-mse-840000-ema-pruned.ckpt
Applying xformers cross attention optimization.
Weights loaded in 15.0s (load weights from disk: 0.2s, apply weights to model: 12.1s, load VAE: 0.5s, move model to device: 2.2s).
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10/10 [00:04<00:00,  2.00it/s]
Loading weights [92970aa785] from C:\Users\user\Desktop\git\stable-diffusion-webui\models\Stable-diffusion\dreamlikePhotoreal20_.safetensors0:04,  2.17it/s]
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 429, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 78, in __call__
    return await self.app(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\fastapi\applications.py", line 273, in __call__
    await super().__call__(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\applications.py", line 122, in __call__
    await self.middleware_stack(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in __call__
    raise exc
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 24, in __call__
    await responder(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\gzip.py", line 44, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 79, in __call__
    raise exc
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 21, in __call__
    raise e
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 718, in __call__
    await route.handle(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\routing.py", line 66, in app
    response = await func(request)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\fastapi\routing.py", line 165, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\starlette\concurrency.py", line 41, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\anyio\to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\anyio\_backends\_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\progress.py", line 85, in progressapi
    shared.state.set_current_image()
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\shared.py", line 250, in set_current_image
    self.do_set_current_image()
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\shared.py", line 260, in do_set_current_image
    self.assign_current_image(modules.sd_samplers.sample_to_image(self.current_latent))
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\sd_samplers_common.py", line 46, in sample_to_image
    return single_sample_to_image(samples[index], approximation)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\sd_samplers_common.py", line 37, in single_sample_to_image
    x_sample = processing.decode_first_stage(shared.sd_model, sample.unsqueeze(0))[0]
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\processing.py", line 423, in decode_first_stage
    x = model.decode_first_stage(x)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\sd_hijack_utils.py", line 17, in <lambda>
    setattr(resolved_obj, func_path[-1], lambda *args, **kwargs: self(*args, **kwargs))
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\modules\sd_hijack_utils.py", line 28, in __call__
    return self.__orig_func(*args, **kwargs)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\torch\autograd\grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\diffusion\ddpm.py", line 826, in decode_first_stage
    return self.first_stage_model.decode(z)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\repositories\stable-diffusion-stability-ai\ldm\models\autoencoder.py", line 89, in decode
    z = self.post_quant_conv(z)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1194, in _call_impl
    return forward_call(*input, **kwargs)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\extensions-builtin\Lora\lora.py", line 201, in lora_Conv2d_forward
    return lora_forward(self, input, torch.nn.Conv2d_forward_before_lora(self, input))
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 463, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "C:\Users\user\Desktop\git\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\conv.py", line 459, in _conv_forward
    return F.conv2d(input, weight, bias, self.stride,
RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

Additional information

No response

PestToast commented 1 year ago

Tried running with xformers disabled, and I get a different runtime error: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument weight in method wrapper__native_group_norm)

Mistborn-First-Era commented 1 year ago

I have the same problem but worse. My vram steadily increases as I switch ckpt until my XYZ plot gets an out of memory error.

myndxero commented 1 year ago

Just a random thought of something to try:

In your Settings, Stable Diffusion section, check and see if you're caching them or not, either the checkpoints or the vae. I leave mine on 0. I had developed some memory issues awhile back with these options enabled. Maybe this will help.

As it were, I've no issue using checkpoints on my Y axis for my grid. The one issue I do have is the max size limited xyz grid imposes. Been trying to figure out how to change/delete it.

Mistborn-First-Era commented 1 year ago

@myndxero I have them at 0 as well. I was also doing this no problem a while ago and have since changed no settings.

As it were, I've no issue using checkpoints on my Y axis for my grid. The one issue I do have is the max size limited xyz grid imposes. Been trying to figure out how to change/delete it.

Use infinity grid extension. It creates an html that you can view so size isn't an issue

myndxero commented 1 year ago

I am aware of infinity grid, however, I like having a proper chart image as well for certain things. I believe I found the culprit maybe. Never noticed the setting for megapixels in the UI, must be somewhat newish maybe. Changed this and testing. I've made massively gigantic grids before, talking about a couple gigabytes big in jpg, so it was also a curiosity about being able to do something again, that I had done before.

On Wed, Mar 15, 2023 at 8:44 PM Mistborn-First-Era @.***> wrote:

@myndxero https://github.com/myndxero I have them at 0 as well. I was also doing this no problem a while ago and have since changed no settings.

As it were, I've no issue using checkpoints on my Y axis for my grid. The one issue I do have is the max size limited xyz grid imposes. Been trying to figure out how to change/delete it.

Use infinity grid extension. It creates an html that you can view so size isn't an issue

— Reply to this email directly, view it on GitHub https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/8656#issuecomment-1471151342, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2755JZ4UBLLHHNFPGBIXGLW4JV6FANCNFSM6AAAAAAV4E7KB4 . You are receiving this because you were mentioned.Message ID: @.***>

PestToast commented 1 year ago

Just a random thought of something to try:

In your Settings, Stable Diffusion section, check and see if you're caching them or not, either the checkpoints or the vae. I leave mine on 0. I had developed some memory issues awhile back with these options enabled. Maybe this will help.

As it were, I've no issue using checkpoints on my Y axis for my grid. The one issue I do have is the max size limited xyz grid imposes. Been trying to figure out how to change/delete it.

Just tried this, and I'm getting the same error message: RuntimeError: Input type (torch.cuda.FloatTensor) and weight type (torch.FloatTensor) should be the same

It's frustrating, because I've definitely been able to do this in the past. It looks like it's some sort of cuda problem. Not sure how I would troubleshoot that, though.

PestToast commented 1 year ago

Also, after the first error message, the rest of the models load (in the commandline) just fine, as they normally should

AUTOMATIC1111 / stable-diffusion-webui