search

Haidra-Org / hordelib_old

A wrapper around ComfyUI to allow use by the AI Horde.
GNU Affero General Public License v3.0
11 stars 0 forks source link

Exception: "RuntimeError: shape '[3072, 768]' is invalid for input of size 4194304" #352

Closed db0 closed 1 year ago

db0 commented 1 year ago

Not sure what caused it but I suspect a bum LoRa. Here's the full log

DEBUG      | 2023-06-01 11:32:44.446929 | worker.jobs.poppers:__init__:254 - {'name': 'The Deep Questioning', 'forms': ['caption', 'nsfw', 'interrogation', '4x_AnimeSharp', 'RealESRGAN_x2plus', 'CodeFormers', 'strip_background', 'RealESRGAN_x4plus', 'NMKD_Siax', 'GFPGAN', 'RealESRGAN_x4plus_anime_6B'], 'amount': 30, 'priority_usernames': ['gruevy#242', 'DeadManWalking#672', 'Yeyosky#1799', 'shinChilla#1938', 'DaigoDawn#3136', 'Mikkos#3260'], 'threads': 20, 'bridge_version': 22, 'bridge_agent': 'AI Horde Worker:22:https://github.com/db0/AI-Horde-Worker', 'max_tiles': 128}
INIT_OK    | 2023-06-01 11:32:44.525944 | hordelib.model_manager.base:load:348 - ICBINP - I Can't Believe It's Not Photography: 6.69 seconds
DEBUG      | 2023-06-01 11:32:44.526313 | worker.workers.framework:check_running_job_status:173 - Job finished successfully in 9.999s (Total Completed: 53)
INFO       | 2023-06-01 11:32:44.550699 | worker.jobs.poppers:__init__:163 - True
DEBUG      | 2023-06-01 11:32:44.815651 | worker.jobs.poppers:horde_pop:40 - Job pop took 0.263189 (node: medusa:7001:4.15.5)
DEBUG      | 2023-06-01 11:32:44.820037 | worker.workers.framework:start_job:142 - New job processing
DEBUG      | 2023-06-01 11:32:44.820147 | worker.jobs.stable_diffusion:start_job:51 - Starting job in threadpool for model: ICBINP - I Can't Believe It's Not Photography
DEBUG      | 2023-06-01 11:32:44.820510 | worker.jobs.stable_diffusion:start_job:61 - ViT-L/14 model loaded
DEBUG      | 2023-06-01 11:32:44.820640 | worker.jobs.stable_diffusion:start_job:148 - txt2img (ICBINP - I Can't Believe It's Not Photography) request with id fd81f27b-20d3-425a-8e8f-24ba7942557d picked up. Initiating work...
INFO       | 2023-06-01 11:32:44.820695 | worker.jobs.stable_diffusion:start_job:166 - Starting generation for id fd81f27b-20d3-425a-8e8f-24ba7942557d: ICBINP - I Can't Believe It's Not Photography @ 512x832 for 40 steps k_euler_a. Prompt length is 239 characters And it appears to contain 1 loras: ['35822']
INFO       | 2023-06-01 11:32:44.820894 | hordelib.comfy_horde:get_pipeline_data:308 - Running pipeline stable_diffusion
DEBUG      | 2023-06-01 11:32:44.821022 | hordelib.horde:_final_pipeline_adjustments:307 - Found valid lora djz Robot Punks
DEBUG      | 2023-06-01 11:32:44.821084 | hordelib.comfy_horde:reconnect_input:446 - Request to reconnect input lora_0.model to output model_loader
DEBUG      | 2023-06-01 11:32:44.821121 | hordelib.comfy_horde:reconnect_input:466 - Request completed to reconnect input lora_0.model to output model_loader
DEBUG      | 2023-06-01 11:32:44.821152 | hordelib.comfy_horde:reconnect_input:446 - Request to reconnect input lora_0.clip to output model_loader
DEBUG      | 2023-06-01 11:32:44.821194 | hordelib.comfy_horde:reconnect_input:466 - Request completed to reconnect input lora_0.clip to output model_loader
DEBUG      | 2023-06-01 11:32:44.821236 | hordelib.comfy_horde:reconnect_input:446 - Request to reconnect input sampler.model to output lora_0
DEBUG      | 2023-06-01 11:32:44.821277 | hordelib.comfy_horde:reconnect_input:466 - Request completed to reconnect input sampler.model to output lora_0
DEBUG      | 2023-06-01 11:32:44.821319 | hordelib.comfy_horde:reconnect_input:446 - Request to reconnect input upscale_sampler.model to output lora_0
DEBUG      | 2023-06-01 11:32:44.821366 | hordelib.comfy_horde:reconnect_input:461 - Attempt to reconnect unknown input upscale_sampler.model
DEBUG      | 2023-06-01 11:32:44.821407 | hordelib.comfy_horde:reconnect_input:446 - Request to reconnect input clip_skip.clip to output lora_0
DEBUG      | 2023-06-01 11:32:44.821448 | hordelib.comfy_horde:reconnect_input:466 - Request completed to reconnect input clip_skip.clip to output lora_0
DEBUG      | 2023-06-01 11:32:44.821495 | hordelib.horde:lock_models:490 - Locked models lora-djz Robot Punks.safetensors,ICBINP - I Can't Believe It's Not Photography
DEBUG      | 2023-06-01 11:32:44.821555 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter loras
DEBUG      | 2023-06-01 11:32:44.821611 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter upscale_sampler.denoise
DEBUG      | 2023-06-01 11:32:44.821650 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter upscale_sampler.seed
DEBUG      | 2023-06-01 11:32:44.821678 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter upscale_sampler.cfg
DEBUG      | 2023-06-01 11:32:44.821705 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter upscale_sampler.steps
DEBUG      | 2023-06-01 11:32:44.821730 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter upscale_sampler.sampler_name
DEBUG      | 2023-06-01 11:32:44.821756 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter controlnet_apply.strength
DEBUG      | 2023-06-01 11:32:44.821782 | hordelib.comfy_horde:_set:429 - Attempt to set unknown pipeline parameter controlnet_model_loader.control_net_name
DEBUG      | 2023-06-01 11:32:44.821807 | hordelib.comfy_horde:_set:437 - Attempt to set parameter CREATED parameter 'model_loader.model_manager'
DEBUG      | 2023-06-01 11:32:44.822436 | hordelib.comfy_horde:send_sync:477 - execution_start, {'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.822521 | hordelib.comfy_horde:send_sync:477 - execution_cached, {'nodes': [], 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.822582 | hordelib.comfy_horde:send_sync:477 - executing, {'node': 'model_loader', 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.822626 | /home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/nodes/node_model_loader:load_checkpoint:32 - Loading model ICBINP - I Can't Believe It's Not Photography through our custom node
DEBUG      | 2023-06-01 11:32:44.822681 | hordelib.comfy_horde:send_sync:477 - executing, {'node': 'lora_0', 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.936699 | hordelib.comfy_horde:send_sync:477 - executing, {'node': 'clip_skip', 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.936897 | hordelib.comfy_horde:send_sync:477 - executing, {'node': 'prompt', 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
ERROR      | 2023-06-01 11:32:44.944406 | execution:execute:252 - Traceback (most recent call last):
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 250, in execute
    recursive_execute(self.server, prompt, self.outputs, x, extra_data, executed, prompt_id)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute
    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute
    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute
    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 129, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 76, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 69, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/nodes.py", line 55, in encode
    return ([[clip.encode(text), {}]], )
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 503, in encode
    return self.encode_from_tokens(tokens)
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 494, in encode_from_tokens
    raise e
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 489, in encode_from_tokens
    self.patcher.patch_model()
  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 370, in patch_model
    weight += (alpha * torch.mm(mat1.flatten(start_dim=1).float(), mat2.flatten(start_dim=1).float())).reshape(weight.shape).type(weight.dtype).to(weight.device)
RuntimeError: shape '[3072, 768]' is invalid for input of size 4194304

DEBUG      | 2023-06-01 11:32:44.944834 | hordelib.comfy_horde:send_sync:477 - execution_error, {'message': 'Traceback (most recent call last):\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 250, in execute\n    recursive_execute(self.server, prompt, self.outputs, x, extra_data, executed, prompt_id)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute\n    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute\n    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 121, in recursive_execute\n    recursive_execute(server, prompt, outputs, input_unique_id, extra_data, executed, prompt_id)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 129, in recursive_execute\n    output_data, output_ui = get_output_data(obj, input_data_all)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 76, in get_output_data\n    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/execution.py", line 69, in map_node_over_list\n    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/nodes.py", line 55, in encode\n    return ([[clip.encode(text), {}]], )\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 503, in encode\n    return self.encode_from_tokens(tokens)\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 494, in encode_from_tokens\n    raise e\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 489, in encode_from_tokens\n    self.patcher.patch_model()\n  File "/home/db0/projects/AI-Horde-Worker/conda/envs/linux/lib/python3.10/site-packages/hordelib/_comfyui/comfy/sd.py", line 370, in patch_model\n    weight += (alpha * torch.mm(mat1.flatten(start_dim=1).float(), mat2.flatten(start_dim=1).float())).reshape(weight.shape).type(weight.dtype).to(weight.device)\nRuntimeError: shape \'[3072, 768]\' is invalid for input of size 4194304\n', 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:44.944980 | hordelib.comfy_horde:send_sync:477 - executing, {'node': None, 'prompt_id': '7f677635-a23a-4ae9-9b53-bcb19e8238d4'}, 7f677635-a23a-4ae9-9b53-bcb19e8238d4
DEBUG      | 2023-06-01 11:32:45.104651 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.104824 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.104872 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc1.alpha
DEBUG      | 2023-06-01 11:32:45.104909 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.104941 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.104972 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc1.lora_down.weight
DEBUG      | 2023-06-01 11:32:45.105000 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105030 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105058 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc1.lora_up.weight
DEBUG      | 2023-06-01 11:32:45.105085 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105112 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105138 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc2.alpha
DEBUG      | 2023-06-01 11:32:45.105165 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105190 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105216 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc2.lora_down.weight
DEBUG      | 2023-06-01 11:32:45.105240 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105264 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105297 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_mlp_fc2.lora_up.weight
DEBUG      | 2023-06-01 11:32:45.105324 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105350 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105375 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_self_attn_k_proj.alpha
DEBUG      | 2023-06-01 11:32:45.105404 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105429 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105453 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_self_attn_k_proj.lora_down.weight
DEBUG      | 2023-06-01 11:32:45.105479 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105503 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105528 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_self_attn_k_proj.lora_up.weight
DEBUG      | 2023-06-01 11:32:45.105553 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105577 | hordelib.utils.ioredirect:replay:32 - 
DEBUG      | 2023-06-01 11:32:45.105601 | hordelib.utils.ioredirect:replay:32 - lora_te_text_model_encoder_layers_12_self_attn_out_proj.alpha
DEBUG      | 2023-06-01 11:32:45.105625 | hordelib.utils.ioredirect:replay:32 - lora key not loaded
DEBUG      | 2023-06-01 11:32:45.105650 | hordelib.utils.ioredirect:replay:32 - ... (31 KB left)
db0 commented 1 year ago

The LoRa they tried to use is https://civitai.com/models/35822 and it is for 2.1. It might be this is causing the conflict. I'll run some tests

db0 commented 1 year ago

Replicated in tests. I'll fix

db0 commented 1 year ago

Fixed in #354