search

lllyasviel / stable-diffusion-webui-forge

GNU Affero General Public License v3.0
8.56k stars 839 forks source link

KeyError: 'gelu_new' 'gelu_new' #1179

Open hok0901 opened 3 months ago

hok0901 commented 3 months ago

ControlNet preprocessor location: G:\webui_forge_cu121_torch231\webui\models\ControlNetPreprocessor 2024-08-16 16:49:29,387 - ControlNet - INFO - ControlNet UI callback registered. Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips_t5xxlfp8.safetensors', 'hash': '3b2e42cf'}, 'additional_modules': [], 'unet_storage_dtype': torch.float8_e4m3fn} Running on local URL: http://127.0.0.1:7860

To create a public link, set share=True in launch(). Startup time: 21.5s (prepare environment: 6.0s, launcher: 1.8s, import torch: 4.5s, initialize shared: 0.7s, other imports: 0.9s, list SD models: 0.2s, load scripts: 1.8s, create ui: 2.4s, gradio launch: 1.0s, add APIs: 2.2s). Environment vars changed: {'stream': False, 'inference_memory': 1024.0, 'pin_shared_memory': False} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips_t5xxlfp8.safetensors', 'hash': '3b2e42cf'}, 'additional_modules': [], 'unet_storage_dtype': None} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips_t5xxlfp8.safetensors', 'hash': '3b2e42cf'}, 'additional_modules': [], 'unet_storage_dtype': torch.float8_e4m3fn} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium.safetensors', 'hash': '9b88d133'}, 'additional_modules': [], 'unet_storage_dtype': torch.float8_e4m3fn} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips.safetensors', 'hash': '080136ef'}, 'additional_modules': [], 'unet_storage_dtype': torch.float8_e4m3fn} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips.safetensors', 'hash': '080136ef'}, 'additional_modules': ['G:\ComfyUI\models\vae\t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': torch.float8_e4m3fn} Model selected: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips.safetensors', 'hash': '080136ef'}, 'additional_modules': ['G:\ComfyUI\models\vae\t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': torch.float8_e4m3fn} Loading Model: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips.safetensors', 'hash': '080136ef'}, 'additional_modules': ['G:\ComfyUI\models\vae\t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': torch.float8_e4m3fn} StateDict Keys: {'unet': 491, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 517, 'text_encoder_3': 220, 'ignore': 0} Using Detected T5 Data Type: torch.float8_e4m3fn Traceback (most recent call last): File "G:\webui_forge_cu121_torch231\webui\modules_forge\main_thread.py", line 30, in work self.result = self.func(*self.args, self.kwargs) File "G:\webui_forge_cu121_torch231\webui\modules\txt2img.py", line 110, in txt2img_function processed = processing.process_images(p) File "G:\webui_forge_cu121_torch231\webui\modules\processing.py", line 789, in process_images p.sd_model, just_reloaded = forge_model_reload() File "C:\Program Files\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "G:\webui_forge_cu121_torch231\webui\modules\sd_models.py", line 501, in forge_model_reload sd_model = forge_loader(state_dict, additional_state_dicts=additional_state_dicts) File "C:\Program Files\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "G:\webui_forge_cu121_torch231\webui\backend\loader.py", line 261, in forge_loader component = load_huggingface_component(estimated_config, component_name, lib_name, cls_name, local_path, component_sd) File "G:\webui_forge_cu121_torch231\webui\backend\loader.py", line 94, in load_huggingface_component model = IntegratedT5(config) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 211, in init self.transformer = T5(config) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 199, in init self.encoder = T5Stack(self.num_layers, model_dim, model_dim, config["d_ff"], config["dense_act_fn"], config["is_gated_act"], config["num_heads"], config["model_type"] != "umt5") File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 172, in init [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)] File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 172, in [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)] File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 159, in init self.layer.append(T5LayerFF(model_dim, ff_dim, ff_activation, gated_act)) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 58, in init self.DenseReluDense = T5DenseGatedActDense(model_dim, ff_dim, ff_activation) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 44, in init self.act = activations[ff_activation] KeyError: 'gelu_new' 'gelu_new' Loading Model: {'checkpoint_info': {'filename': 'G:\ComfyUI\models\checkpoints\SD3\sd3_medium_incl_clips.safetensors', 'hash': '080136ef'}, 'additional_modules': ['G:\ComfyUI\models\vae\t5xxl_fp8_e4m3fn.safetensors'], 'unet_storage_dtype': torch.float8_e4m3fn} StateDict Keys: {'unet': 491, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 517, 'text_encoder_3': 220, 'ignore': 0} Using Detected T5 Data Type: torch.float8_e4m3fn Traceback (most recent call last): File "G:\webui_forge_cu121_torch231\webui\modules_forge\main_thread.py", line 30, in work self.result = self.func(*self.args, self.kwargs) File "G:\webui_forge_cu121_torch231\webui\modules\txt2img.py", line 110, in txt2img_function processed = processing.process_images(p) File "G:\webui_forge_cu121_torch231\webui\modules\processing.py", line 789, in process_images p.sd_model, just_reloaded = forge_model_reload() File "C:\Program Files\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(*args, *kwargs) File "G:\webui_forge_cu121_torch231\webui\modules\sd_models.py", line 501, in forge_model_reload sd_model = forge_loader(state_dict, additional_state_dicts=additional_state_dicts) File "C:\Program Files\Python310\lib\site-packages\torch\utils_contextlib.py", line 115, in decorate_context return func(args, kwargs) File "G:\webui_forge_cu121_torch231\webui\backend\loader.py", line 261, in forge_loader component = load_huggingface_component(estimated_config, component_name, lib_name, cls_name, local_path, component_sd) File "G:\webui_forge_cu121_torch231\webui\backend\loader.py", line 94, in load_huggingface_component model = IntegratedT5(config) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 211, in init self.transformer = T5(config) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 199, in init self.encoder = T5Stack(self.num_layers, model_dim, model_dim, config["d_ff"], config["dense_act_fn"], config["is_gated_act"], config["num_heads"], config["model_type"] != "umt5") File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 172, in init [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)] File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 172, in [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)] File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 159, in init self.layer.append(T5LayerFF(model_dim, ff_dim, ff_activation, gated_act)) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 58, in init self.DenseReluDense = T5DenseGatedActDense(model_dim, ff_dim, ff_activation) File "G:\webui_forge_cu121_torch231\webui\backend\nn\t5.py", line 44, in init self.act = activations[ff_activation] KeyError: 'gelu_new' 'gelu_new'

adde88 commented 3 months ago

I ran into this issue as well, so i started testing several commits while doing git bisect. It seemed to indicate that the issue started at commit 9bc2d04ca942d7d71d1f4c4c2e94945ecdf0e4a5:

9bc2d04ca942d7d71d1f4c4c2e94945ecdf0e4a5 is the first bad commit
commit 9bc2d04ca942d7d71d1f4c4c2e94945ecdf0e4a5
Merge: d7151b4d 9f5a27ca
Author: Panchovix <francisco.jaqueat@gmail.com>
Date:   Mon Aug 19 16:58:52 2024 -0400

    Merge pull request #1310 from lllyasviel/more_schedulers

    Add Align Your Steps GITS, AYS 11 Steps and AYS 32 Steps Schedulers.

 modules/sd_schedulers.py | 76 ++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 76 insertions(+)
binarynoise commented 2 months ago

for me, bb58520a4c5bf268b567a6f7cb58901bd942ae59 is the first bad one, before I get the 'NoneType' object is not iterable

BellemareMederic commented 2 months ago

same problem here

MichaelJ9000 commented 2 months ago

you shoud enabel Settings->Stable Diffusion3->Enable T5

GlamoramaAttack commented 2 months ago

Is SD3 currently working for you in Forge? If so, maybe you share which Forge version number you use.

'Enable T5' - no matter whether enabled or not (!)* - delivers in my setup the same error message at the end as OP's, the "t5.py" script is the one that throws an error in the "activations" part (the last lines), everything seems to be fine until...

self.DenseReluDense = T5DenseGatedActDense(model_dim, ff_dim, ff_activation)

File "D:\sd-forge\backend\nn\t5.py", line 44, in init self.act = activations[ff_activation] KeyError: 'gelu_new' 'gelu_new'

I'm on version: f2.0.1v1.10.1-previous-528-ge55cde9b (three days old, Python 3.10.11, torch: 2.3.1+cu121), had the same problem with the version before, but don't recall exactly when I first noticed it.

What happens after line 44 in "t.5.py"? I don't have a clue but the script has another 168 lines. The script continues next with... 

def forward(self, x):
    hidden_gelu = self.act(self.wi_0(x))
    hidden_linear = self.wi_1(x)
    x = hidden_gelu * hidden_linear
    x = self.wo(x)
    return x

*= And if T5 is enabled in the SD3 setting it first delivers a series of error messages:

ERROR: Exception in ASGI application Traceback (most recent call last): File "D:\sd-forge\venv\lib\site-packages\starlette\middleware\exceptions.py", line 68, in call await self.app(scope, receive, sender) File "D:\sd-forge\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 20, in call raise e File "D:\sd-forge\venv\lib\site-packages\fastapi\middleware\asyncexitstack.py", line 17, in call await self.app(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\routing.py", line 718, in call await route.handle(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\routing.py", line 276, in handle await self.app(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\routing.py", line 69, in app await response(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\responses.py", line 270, in call async with anyio.create_task_group() as task_group: File "D:\sd-forge\venv\lib\site-packages\anyio_backends_asyncio.py", line 597, in aexit raise exceptions[0] File "D:\sd-forge\venv\lib\site-packages\starlette\responses.py", line 273, in wrap await func() File "D:\sd-forge\venv\lib\site-packages\starlette\responses.py", line 262, in stream_response async for chunk in self.body_iterator: File "D:\sd-forge\venv\lib\site-packages\gradio\routes.py", line 956, in sse_stream raise e File "D:\sd-forge\venv\lib\site-packages\gradio\routes.py", line 891, in sse_stream raise HTTPException( fastapi.exceptions.HTTPException

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "D:\sd-forge\venv\lib\site-packages\uvicorn\protocols\http\h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] File "D:\sd-forge\venv\lib\site-packages\uvicorn\middleware\proxy_headers.py", line 70, in call return await self.app(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\fastapi\applications.py", line 1106, in call await super().call(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\applications.py", line 122, in call await self.middleware_stack(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\middleware\errors.py", line 184, in call raise exc File "D:\sd-forge\venv\lib\site-packages\starlette\middleware\errors.py", line 162, in call await self.app(scope, receive, _send) File "D:\sd-forge\venv\lib\site-packages\gradio\route_utils.py", line 724, in call await self.app(scope, receive, send) File "D:\sd-forge\venv\lib\site-packages\starlette\middleware\exceptions.py", line 83, in call raise RuntimeError(msg) from exc RuntimeError: Caught handled exception, but response already started.

webmaster-exit-1 commented 2 months ago

I'm getting this error as well.

StateDict Keys: {'transformer': 780, 'vae': 244, 'text_encoder': 196, 'text_encoder_2': 220, 'ignore': 0}
Using Default T5 Data Type: torch.float16
Traceback (most recent call last):
  File "/var/home/$USER/stable-diffusion-webui-forge/modules_forge/main_thread.py", line 30, in work
    self.result = self.func(*self.args, **self.kwargs)
  File "/var/home/$USER/stable-diffusion-webui-forge/extensions/sd-webui-fsr-integration/modules/txt2img.py", line 123, in txt2img_function
    processed = processing.process_images(p)
  File "/var/home/$USER/stable-diffusion-webui-forge/modules/processing.py", line 800, in process_images
    p.sd_model, just_reloaded = forge_model_reload()
  File "/var/home/$USER/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/var/home/$USER/stable-diffusion-webui-forge/modules/sd_models.py", line 494, in forge_model_reload
    sd_model = forge_loader(state_dict, additional_state_dicts=additional_state_dicts)
  File "/var/home/$USER/stable-diffusion-webui-forge/venv/lib64/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/loader.py", line 283, in forge_loader
    component = load_huggingface_component(estimated_config, component_name, lib_name, cls_name, local_path, component_sd)
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/loader.py", line 103, in load_huggingface_component
    model = IntegratedT5(config)
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 211, in __init__
    self.transformer = T5(config)
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 199, in __init__
    self.encoder = T5Stack(self.num_layers, model_dim, model_dim, config["d_ff"], config["dense_act_fn"], config["is_gated_act"], config["num_heads"], config["model_type"] != "umt5")
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 172, in __init__
    [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)]
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 172, in <listcomp>
    [T5Block(model_dim, inner_dim, ff_dim, ff_activation, gated_act, num_heads, relative_attention_bias=((not relative_attention) or (i == 0))) for i in range(num_layers)]
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 159, in __init__
    self.layer.append(T5LayerFF(model_dim, ff_dim, ff_activation, gated_act))
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 58, in __init__
    self.DenseReluDense = T5DenseGatedActDense(model_dim, ff_dim, ff_activation)
  File "/var/home/$USER/stable-diffusion-webui-forge/backend/nn/t5.py", line 44, in __init__
    self.act = activations[ff_activation]
KeyError: 'gelu_new'
'gelu_new'

So Flux is pretty much completely broken atm using this project.

GlamoramaAttack commented 3 weeks ago

With the current version: f2.0.1v1.10.1-previous-587-g1363999f and @graemeniedermayer 's sd3.5 integration at https://github.com/lllyasviel/stable-diffusion-webui-forge/pull/2183 I'm able to use SD3 again, can use 3.5 too.

(I had no problems with Flux in Forge so it was never broken for me).