Prompt validation fails with partially working prompt, leading to stuck main thread

[jkrauss82]

Expected Behavior

Prompt validation should fail and return 400 to client as usual

Actual Behavior

Seeing this stack trace and UI is not reacting anymore. Prompt worker is still running and receiving requests but nothing else works. See debug logs for stack trace

Steps to Reproduce

see the python code below and the attached workflow for a minimal example to reproduce the error.

we delete the one checkpoint loader in the json, then submit it leading to the observed state

import json
import sys
from urllib import request

workflow = "workflow_api_min_example.json"

prompt = None
with open(workflow) as f:
    prompt = json.load(f)
if prompt == None:
    print("Error, could not read workflow from file, abort.")
    sys.exit(1)

def queue_prompt(prompt):
    p = {"prompt": prompt}
    data = json.dumps(p).encode('utf-8')
    req =  request.Request("http://127.0.0.1:7860/prompt", data=data)
    request.urlopen(req)

del prompt["4"]

queue_prompt(prompt)

workflow_api_min_example.json

Debug Logs

$ ./start.sh
Total VRAM 16081 MB, total RAM 31952 MB
pytorch version: 2.4.0+cu121
Set vram state to: NORMAL_VRAM
Device: cuda:0 NVIDIA GeForce RTX 4060 Ti : native
Using pytorch cross attention
[Prompt Server] web root: /home/user/workspace/ComfyUI/web
/home/user/workspace/ComfyUI/venv/lib/python3.10/site-packages/kornia/feature/lightglue.py:44: FutureWarning: `torch.cuda.amp.custom_fwd(args...)` is deprecated. Please use `torch.amp.custom_fwd(args..., device_type='cuda')` instead.
@torch.cuda.amp.custom_fwd(cast_inputs=torch.float32)
Skipping loading of custom nodes
Starting server

To see the GUI go to: http://0.0.0.0:7860
got prompt
Failed to validate prompt for output 10:
* LoraLoader 11:
- Exception when validating inner node: '4'
* VAEDecode 8:
- Exception when validating inner node: '4'
Output will be ignored
Exception in thread Thread-1 (prompt_worker):
Traceback (most recent call last):
File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
self.run()
File "/usr/lib/python3.10/threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "/home/user/workspace/ComfyUI/main.py", line 121, in prompt_worker
e.execute(item[2], prompt_id, item[3], item[4])
File "/home/user/workspace/ComfyUI/execution.py", line 468, in execute
cache.set_prompt(dynamic_prompt, prompt.keys(), is_changed_cache)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 136, in set_prompt
self.cache_key_set = self.key_class(dynprompt, node_ids, is_changed_cache)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 68, in __init__
self.add_keys(node_ids)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 78, in add_keys
self.keys[node_id] = self.get_node_signature(self.dynprompt, node_id)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 83, in get_node_signature
ancestors, order_mapping = self.get_ordered_ancestry(dynprompt, node_id)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 111, in get_ordered_ancestry
self.get_ordered_ancestry_internal(dynprompt, node_id, ancestors, order_mapping)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 123, in get_ordered_ancestry_internal
self.get_ordered_ancestry_internal(dynprompt, ancestor_id, ancestors, order_mapping)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 123, in get_ordered_ancestry_internal
self.get_ordered_ancestry_internal(dynprompt, ancestor_id, ancestors, order_mapping)
File "/home/user/workspace/ComfyUI/comfy_execution/caching.py", line 115, in get_ordered_ancestry_internal
inputs = dynprompt.get_node(node_id)["inputs"]
File "/home/user/workspace/ComfyUI/comfy_execution/graph.py", line 28, in get_node
raise NodeNotFoundError(f"Node {node_id} not found")
comfy_execution.graph.NodeNotFoundError: Node 4 not found

Other

No response

[frankchieng]

API in main.py prompt_worker have bug either

[frankchieng]

bg_change_workflow_api.json @guill @comfyanonymous @mcmonkey4eva the API exception resulting in server lockup happened again with the following json workflow { "4": { "inputs": { "ckpt_name": "juggernautXL_juggernautX.safetensors" }, "class_type": "CheckpointLoaderSimple", "_meta": { "title": "Load Checkpoint" } }, "6": { "inputs": { "text": "fine details, official art, very detailed 8k wallpaper,", "clip": [ "277", 1 ] }, "class_type": "CLIPTextEncode", "_meta": { "title": "CLIP Text Encode (Prompt)" } }, "7": { "inputs": { "text": "asymmetry,lowres, low quality, worst quality, (text:1.2), watermark, painting, drawing, illustration, glitch, deformed, mutated, cross-eyed, ugly,(nsfw:1.2),bare breast,text, watermark", "clip": [ "4", 1 ] }, "class_type": "CLIPTextEncode", "_meta": { "title": "CLIP Text Encode (Prompt)" } }, "48": { "inputs": { "image": "man.jpg", "upload": "image" }, "class_type": "LoadImage", "_meta": { "title": "Load Image" } }, "107": { "inputs": { "model": [ "277", 0 ], "patch": [ "108", 0 ], "latent": [ "253", 0 ] }, "class_type": "INPAINT_ApplyFooocusInpaint", "_meta": { "title": "Apply Fooocus Inpaint" } }, "108": { "inputs": { "head": "fooocus_inpaint_head.pth", "patch": "inpaint_v26.fooocus.patch" }, "class_type": "INPAINT_LoadFooocusInpaint", "_meta": { "title": "Load Fooocus Inpaint" } }, "130": { "inputs": { "seed": 681194540782487, "steps": 30, "cfg": 8, "sampler_name": "euler_ancestral", "scheduler": "karras", "denoise": 1, "model": [ "107", 0 ], "positive": [ "264", 0 ], "negative": [ "264", 1 ], "latent_image": [ "264", 2 ] }, "class_type": "KSampler", "_meta": { "title": "KSampler" } }, "131": { "inputs": { "samples": [ "130", 0 ], "vae": [ "4", 2 ] }, "class_type": "VAEDecode", "_meta": { "title": "VAE Decode" } }, "253": { "inputs": { "grow_mask_by": 1, "pixels": [ "48", 0 ], "vae": [ "4", 2 ], "mask": [ "275", 0 ] }, "class_type": "VAEEncodeForInpaint", "_meta": { "title": "VAE Encode (for Inpainting)" } }, "264": { "inputs": { "positive": [ "6", 0 ], "negative": [ "7", 0 ], "vae": [ "4", 2 ], "pixels": [ "48", 0 ], "mask": [ "275", 0 ] }, "class_type": "InpaintModelConditioning", "_meta": { "title": "InpaintModelConditioning" } }, "272": { "inputs": { "filename_prefix": "ComfyUI", "images": [ "131", 0 ] }, "class_type": "SaveImage", "_meta": { "title": "Save Image" } }, "273": { "inputs": { "rmbgmodel": [ "274", 0 ], "image": [ "48", 0 ] }, "class_type": "BRIA_RMBG_Zho", "_meta": { "title": "BRIA RMBG" } }, "274": { "inputs": {}, "class_type": "BRIA_RMBG_ModelLoader_Zho", "_meta": { "title": "BRIA_RMBG Model Loader" } }, "275": { "inputs": { "mask": [ "273", 1 ] }, "class_type": "InvertMask", "_meta": { "title": "InvertMask" } }, "277": { "inputs": { "lora_name": "XL_add_details.safetensors", "strength_model": 1, "strength_clip": 1, "model": [ "4", 0 ], "clip": [ "4", 1 ] }, "class_type": "LoraLoader", "_meta": { "title": "Load LoRA" } } }, i mean it ran well via default UI, but when switch to API, just failed as my previous bug report,

when i print the input_keys = sorted(inputs.keys()) in def get_ordered_ancestry_internal(self, dynprompt, node_id, ancestors, order_mapping):

every time with different input nodes dictionary{'ckpt_name': 'juggernautXL_juggernautX.safetensors'} <class 'dict'> {'text': 'fine details, official art, very detailed 8k wallpaper,Blue background wall,', 'clip': ['277', 1]} <class 'dict'> {'lora_name': 'XL_add_details.safetensors', 'strength_model': 1.0, 'strength_clip': 1.0, 'model': ['4', 0], 'clip': ['4', 1]} <class 'dict'> {'ckpt_name': 'juggernautXL_juggernautX.safetensors'} <class 'dict'> {'text': 'asymmetry,lowres, low quality, worst quality, (text:1.2), watermark, painting, drawing, illustration, glitch, deformed, mutated, cross-eyed, ugly,(nsfw:1.2),bare breast,text, watermark', 'clip': ['4', 1]} <class 'dict'> {'ckpt_name': 'juggernautXL_juggernautX.safetensors'} <class 'dict'> {'image': '1727524924_35031.png', 'upload': 'image'} <class 'dict'> {'model': ['277', 0], 'patch': ['108', 0], 'latent': ['253', 0]} <class 'dict'> {'grow_mask_by': 1, 'pixels': ['48', 0], 'vae': ['4', 2], 'mask': ['275', 0]} <class 'dict'> {'mask': ['273', 1]} <class 'dict'> {'rmbgmodel': ['274', 0], 'image': ['48', 0]} <class 'dict'> {'image': '1727524924_35031.png', 'upload': 'image'} <class 'dict'> [] <class 'list'> {'ckpt_name': 'juggernautXL_juggernautX.safetensors'} <class 'dict'> {'lora_name': 'XL_add_details.safetensors', 'strength_model': 1.0, 'strength_clip': 1.0, 'model': ['4', 0], 'clip': ['4', 1]} <class 'dict'> {'head': 'fooocus_inpaint_head.pth', 'patch': 'inpaint_v26.fooocus.patch'} <class 'dict'>

you can see one of them is a blank list,not all of them are dict

for current workaround,i just added if isinstance(inputs, dict) in both def get_immediate_node_signature(self, dynprompt, node_id, ancestor_order_mapping): and def get_ordered_ancestry(self, dynprompt, node_id):

[guill]

@frankchieng I'm not able to reproduce this issue on master. The workflow you posted works fine for me, both within the default UI and submitting via the raw API via curl. If you have a workflow that's working in the default UI but not when saved as an API, the most likely possibility is that there's a node relying on extra_pnginfo, though I don't see anything like that here.