search

ltdrdata / ComfyUI-extension-tutorials

584 stars 33 forks source link

How to load properlt Onnx Model? #27

Closed VeteranXT closed 10 months ago

VeteranXT commented 10 months ago

Title says it all? There is no "load Onnx" Loading Stable Diffusion directml with olive works fine. It generates picture around 15s for 512x512. Without onnx it generates about 86.69/s for 512x512.

Loading same model and same prompt it generates quite slowly 76.69-95/s per 512x512. I was tying to make onnx work here using same stuff but somehow i can't seems to make it work.

This is error i get in console. 2024-01-07 18:00:24.4739806 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.4779075 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5033458 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5057012 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5218233 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5241067 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5398494 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5422907 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5576465 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5600128 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5763688 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5786802 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.5943569 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.5967344 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.6124303 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.6148976 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.6303633 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.6327471 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' 2024-01-07 18:00:24.6484582 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_2' 2024-01-07 18:00:24.6507739 [W:onnxruntime:, constant_folding.cc:212 onnxruntime::ConstantFolding::ApplyImpl] Could not find a CPU kernel and hence can't constant fold Clip node '/Clip_1' ONNXDetector: unable to execute. Required inputs (['images']) are missing from input feed (['clip_input']). ERROR:root:!!! Exception during processing !!! ERROR:root:Traceback (most recent call last): File "E:\ComfyUI\execution.py", line 154, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\ComfyUI\execution.py", line 84, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\ComfyUI\execution.py", line 77, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "E:\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\detectors.py", line 97, in doit segs, _ = segs_nodes.SEGSLabelFilter.filter(segs, labels) File "E:\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\segs_nodes.py", line 428, in filter return (segs, (segs[0], []), ) TypeError: 'NoneType' object is not subscriptable

This is in ComfyUI `Error occurred when executing BboxDetectorSEGS:

'NoneType' object is not subscriptable

File "E:\ComfyUI\execution.py", line 154, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) File "E:\ComfyUI\execution.py", line 84, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) File "E:\ComfyUI\execution.py", line 77, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) File "E:\ComfyUI\customnodes\ComfyUI-Impact-Pack\modules\impact\detectors.py", line 97, in doit segs, = segs_nodes.SEGSLabelFilter.filter(segs, labels) File "E:\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\segs_nodes.py", line 428, in filter return (segs, (segs[0], []), )`

this is layout { "last_node_id": 8, "last_link_id": 12, "nodes": [ { "id": 2, "type": "ONNXDetectorProvider", "pos": [ 37, 22 ], "size": [ 680.4002042968752, 58 ], "flags": {}, "order": 0, "mode": 0, "outputs": [ { "name": "BBOX_DETECTOR", "type": "BBOX_DETECTOR", "links": [ 1 ], "shape": 3 } ], "properties": { "Node name for S&R": "ONNXDetectorProvider" }, "widgets_values": [ "fantassified_icons_v2_OP\\safety_checker\\model.onnx" ] }, { "id": 4, "type": "PreviewImage", "pos": [ 1913.7442401123049, 111.14684814453128 ], "size": { "0": 210, "1": 26 }, "flags": {}, "order": 7, "mode": 0, "inputs": [ { "name": "images", "type": "IMAGE", "link": 3 } ], "properties": { "Node name for S&R": "PreviewImage" } }, { "id": 1, "type": "BboxDetectorSEGS", "pos": [ 766, 46 ], "size": { "0": 400, "1": 212.00001525878906 }, "flags": {}, "order": 3, "mode": 0, "inputs": [ { "name": "bbox_detector", "type": "BBOX_DETECTOR", "link": 1, "slot_index": 0 }, { "name": "image", "type": "IMAGE", "link": 5, "slot_index": 1 }, { "name": "detailer_hook", "type": "DETAILER_HOOK", "link": null } ], "outputs": [ { "name": "SEGS", "type": "SEGS", "links": [ 2 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "BboxDetectorSEGS" }, "widgets_values": [ 0.5, 10, 3, 10, "all" ] }, { "id": 5, "type": "EmptyImage", "pos": [ 422, 288 ], "size": { "0": 315, "1": 130 }, "flags": {}, "order": 1, "mode": 0, "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 4, 5 ], "shape": 3 } ], "properties": { "Node name for S&R": "EmptyImage" }, "widgets_values": [ 512, 512, 1, 0 ] }, { "id": 6, "type": "CheckpointLoaderSimple", "pos": [ 72, 378 ], "size": { "0": 315, "1": 98 }, "flags": {}, "order": 2, "mode": 0, "outputs": [ { "name": "MODEL", "type": "MODEL", "links": [ 6 ], "shape": 3 }, { "name": "CLIP", "type": "CLIP", "links": [ 7, 8, 9 ], "shape": 3, "slot_index": 1 }, { "name": "VAE", "type": "VAE", "links": [ 12 ], "shape": 3 } ], "properties": { "Node name for S&R": "CheckpointLoaderSimple" }, "widgets_values": [ "fantassified_icons_v2.safetensors" ] }, { "id": 7, "type": "CLIPTextEncode", "pos": [ 689, 464 ], "size": { "0": 400, "1": 200 }, "flags": {}, "order": 4, "mode": 0, "inputs": [ { "name": "clip", "type": "CLIP", "link": 7 } ], "outputs": [ { "name": "CONDITIONING", "type": "CONDITIONING", "links": [ 10 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "CLIPTextEncode" }, "widgets_values": [ "" ] }, { "id": 8, "type": "CLIPTextEncode", "pos": [ 696.4842401123047, 725.8268481445314 ], "size": { "0": 400, "1": 200 }, "flags": {}, "order": 5, "mode": 0, "inputs": [ { "name": "clip", "type": "CLIP", "link": 8 } ], "outputs": [ { "name": "CONDITIONING", "type": "CONDITIONING", "links": [ 11 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "CLIPTextEncode" }, "widgets_values": [ "" ] }, { "id": 3, "type": "DetailerForEach", "pos": [ 1364.404220581055, 95.41687011718756 ], "size": { "0": 400, "1": 552 }, "flags": {}, "order": 6, "mode": 0, "inputs": [ { "name": "image", "type": "IMAGE", "link": 4, "slot_index": 0 }, { "name": "segs", "type": "SEGS", "link": 2 }, { "name": "model", "type": "MODEL", "link": 6, "slot_index": 2 }, { "name": "clip", "type": "CLIP", "link": 9 }, { "name": "vae", "type": "VAE", "link": 12, "slot_index": 4 }, { "name": "positive", "type": "CONDITIONING", "link": 10 }, { "name": "negative", "type": "CONDITIONING", "link": 11 }, { "name": "detailer_hook", "type": "DETAILER_HOOK", "link": null } ], "outputs": [ { "name": "IMAGE", "type": "IMAGE", "links": [ 3 ], "shape": 3, "slot_index": 0 } ], "properties": { "Node name for S&R": "DetailerForEach" }, "widgets_values": [ 384, true, 1024, 187579551106667, "randomize", 20, 8, "euler", "normal", 0.5, 5, true, true, "", 1 ] } ], "links": [ [ 1, 2, 0, 1, 0, "BBOX_DETECTOR" ], [ 2, 1, 0, 3, 1, "SEGS" ], [ 3, 3, 0, 4, 0, "IMAGE" ], [ 4, 5, 0, 3, 0, "IMAGE" ], [ 5, 5, 0, 1, 1, "IMAGE" ], [ 6, 6, 0, 3, 2, "MODEL" ], [ 7, 6, 1, 7, 0, "CLIP" ], [ 8, 6, 1, 8, 0, "CLIP" ], [ 9, 6, 1, 3, 3, "CLIP" ], [ 10, 7, 0, 3, 5, "CONDITIONING" ], [ 11, 8, 0, 3, 6, "CONDITIONING" ], [ 12, 6, 2, 3, 4, "VAE" ] ], "groups": [], "config": {}, "extra": {}, "version": 0.4 }

Using ComfyUI-Impact pack for this and nothing else.

ltdrdata commented 10 months ago

That node doesn't support olive.

It's designed to support detection models in ONNX format. Although it was created upon request, it's a node with very limited utility, and currently, it is not aware of compatible models other than the nudenet model.