search

huchenlei / stable-diffusion-webui

Stable Diffusion web UI
GNU Affero General Public License v3.0
2 stars 0 forks source link

[Bug]: Unable to load model after applying PR#15821 when FP8 weight is enabled #9

Open serick4126 opened 1 month ago

serick4126 commented 1 month ago

Is there an existing issue for this?

What happened?

The model failed to load because FP8 weight was set to enabled in the settings optimization. image

When I disabled it, the model loaded without problems.

Steps to reproduce the problem

  1. Enable FP8 weight in the settings optimization
  2. Load a checkpoint model
  3. Runtime error occurs and error message is displayed on the console "RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float8_e4m3fn"

What should have happened?

Need to guide to disable FP8 weight.

Commit where the problem happens

https://github.com/AUTOMATIC1111/stable-diffusion-webui/commit/eff3d791ff562a86b673a226f2902283f49cd2ab

What platforms do you use to access the UI ?

Windows

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

No

List of extensions

No

Console logs

loading stable diffusion model: RuntimeError
Traceback (most recent call last):
  File "C:\Python\Python310\lib\threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "C:\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Python\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "F:\stable-diffusion-webui-dev\modules\initialize.py", line 149, in load_model
    shared.sd_model  # noqa: B018
  File "F:\stable-diffusion-webui-dev\modules\shared_items.py", line 175, in sd_model
    return modules.sd_models.model_data.get_sd_model()
  File "F:\stable-diffusion-webui-dev\modules\sd_models.py", line 636, in get_sd_model
    load_model()
  File "F:\stable-diffusion-webui-dev\modules\sd_models.py", line 787, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
  File "F:\stable-diffusion-webui-dev\modules\sd_models.py", line 671, in get_empty_cond
    d = sd_model.get_learned_conditioning([""])
  File "F:\stable-diffusion-webui-dev\modules\sd_models_xl.py", line 32, in get_learned_conditioning
    c = self.conditioner(sdxl_conds, force_zero_embeddings=['txt'] if force_zero_negative_prompt else [])
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\repositories\generative-models\sgm\modules\encoders\modules.py", line 141, in forward
    emb_out = embedder(batch[embedder.input_key])
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\modules\sd_hijack_clip.py", line 234, in forward
    z = self.process_tokens(tokens, multipliers)
  File "F:\stable-diffusion-webui-dev\modules\sd_hijack_clip.py", line 276, in process_tokens
    z = self.encode_with_transformers(tokens)
  File "F:\stable-diffusion-webui-dev\modules\sd_hijack_clip.py", line 354, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=self.wrapped.layer == "hidden")
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 822, in forward
    return self.text_model(
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 740, in forward
    encoder_outputs = self.encoder(
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 654, in forward
    layer_outputs = encoder_layer(
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 383, in forward
    hidden_states, attn_weights = self.self_attn(
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 272, in forward
    query_states = self.q_proj(hidden_states) * self.scale
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "F:\stable-diffusion-webui-dev\extensions-builtin\Lora\networks.py", line 508, in network_Linear_forward
    return originals.Linear_forward(self, input)
  File "F:\stable-diffusion-webui-dev\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float8_e4m3fn

Stable diffusion model failed to load

Additional information

No response

Serkodim commented 1 month ago

I have the same error Hardware: RTX 4050 laptop 6GB, 16GB RAM COMMANDLINE_ARGS=--xformers --disable-all-extensions --medvram --precision half also the same error with --medvram-sdxl FP8 enable Log: 

venv "C:\Users\Balab\stable-diffusion-webui\venv\Scripts\Python.exe"
Python 3.10.6 (tags/v3.10.6:9c7b4bd, Aug  1 2022, 21:53:49) [MSC v.1932 64 bit (AMD64)]
Version: v1.9.3-13-g8e355f08
Commit hash: 8e355f08b2e9f6b2e75581e5da752bb428c4bcdd
Launching Web UI with arguments: --xformers --medvram-sdxl --disable-all-extensions --precision half
ldm/sgm GroupNorm32 replaced with normal torch.nn.GroupNorm due to `--precision half`.
*** "--disable-all-extensions" arg was used, will not load any extensions ***
Loading weights [8421598e93] from C:\Users\Balab\stable-diffusion-webui\models\Stable-diffusion\Anything_XL.safetensors
Running on local URL:  http://127.0.0.1:7860

To create a public link, set `share=True` in `launch()`.
Creating model from config: C:\Users\Balab\stable-diffusion-webui\repositories\generative-models\configs\inference\sd_xl_base.yaml
Startup time: 19.9s (prepare environment: 6.5s, import torch: 6.0s, import gradio: 1.6s, setup paths: 2.2s, initialize shared: 0.3s, other imports: 1.1s, load scripts: 0.9s, create ui: 0.5s, gradio launch: 0.6s).
Applying attention optimization: xformers... done.
loading stable diffusion model: RuntimeError
Traceback (most recent call last):
  File "C:\Users\Balab\AppData\Local\Programs\Python\Python310\lib\threading.py", line 973, in _bootstrap
    self._bootstrap_inner()
  File "C:\Users\Balab\AppData\Local\Programs\Python\Python310\lib\threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "C:\Users\Balab\AppData\Local\Programs\Python\Python310\lib\threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\modules\initialize.py", line 149, in load_model
    shared.sd_model  # noqa: B018
  File "C:\Users\Balab\stable-diffusion-webui\modules\shared_items.py", line 175, in sd_model
    return modules.sd_models.model_data.get_sd_model()
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_models.py", line 636, in get_sd_model
    load_model()
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_models.py", line 787, in load_model
    sd_model.cond_stage_model_empty_prompt = get_empty_cond(sd_model)
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_models.py", line 671, in get_empty_cond
    d = sd_model.get_learned_conditioning([""])
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_models_xl.py", line 32, in get_learned_conditioning
    c = self.conditioner(sdxl_conds, force_zero_embeddings=['txt'] if force_zero_negative_prompt else [])
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1568, in _call_impl
    result = forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\repositories\generative-models\sgm\modules\encoders\modules.py", line 141, in forward
    emb_out = embedder(batch[embedder.input_key])
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_hijack_clip.py", line 234, in forward
    z = self.process_tokens(tokens, multipliers)
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_hijack_clip.py", line 276, in process_tokens
    z = self.encode_with_transformers(tokens)
  File "C:\Users\Balab\stable-diffusion-webui\modules\sd_hijack_clip.py", line 354, in encode_with_transformers
    outputs = self.wrapped.transformer(input_ids=tokens, output_hidden_states=self.wrapped.layer == "hidden")
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 822, in forward
    return self.text_model(
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 740, in forward
    encoder_outputs = self.encoder(
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 654, in forward
    layer_outputs = encoder_layer(
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 383, in forward
    hidden_states, attn_weights = self.self_attn(
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\transformers\models\clip\modeling_clip.py", line 272, in forward
    query_states = self.q_proj(hidden_states) * self.scale
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "C:\Users\Balab\stable-diffusion-webui\venv\lib\site-packages\torch\nn\modules\linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 must have the same dtype, but got Half and Float8_e4m3fn

Stable diffusion model failed to load