search

Nuked88 / ComfyUI-N-Nodes

A suite of custom nodes for ConfyUI that includes GPT text-prompt generation, LoadVideo, SaveVideo, LoadFramesFromFolder and FrameInterpolator
MIT License
205 stars 22 forks source link

ERROR segment anything #9

Closed t8ja closed 1 year ago

t8ja commented 1 year ago

VIDEO input NODE

Error occurred when executing GroundingDinoSAMSegment (segment anything):

cannot unpack non-iterable NoneType object

  File "/mnt/starT/AI/ComfyUI/execution.py", line 152, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/starT/AI/ComfyUI/execution.py", line 82, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/starT/AI/ComfyUI/execution.py", line 75, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/starT/AI/ComfyUI/custom_nodes/comfyui_segment_anything/node.py", line 305, in main
    (images, masks) = sam_segment(
    ^^^^^^^^^^^^^^^

FRAME input NODE

`Error occurred when executing GroundingDinoSAMSegment (segment anything):

list index out of range

Last line is different


    File "/mnt/starT/AI/ComfyUI/execution.py", line 59, in slice_dict
    d_new[k] = v[i if len(v) > i else -1]
               ~^^^^^^^^^^^^^^^^^^^^^^^^^
Nuked88 commented 1 year ago

probably GroundingDinoSAMSegment does not support batch? Try (only for test because i cannot do anything about it) using only an image putting images_limit to 1 and batch to 0

Nuked88 commented 1 year ago

uhm i don't know man maybe you can ask to the author of that node now that is clear that is not some bug of mine since even WAS give the same problem

t8ja commented 1 year ago

No, no, your nodes never worked properly (but produced no ERRORS). I only got them to output 1 image at a time (including the preview video consisting of 1 frame).

WAS worked most of the time, until VRAM started getting low so sometimes it wouldn't render a few starting frames and then it would continue till the end, or fail due to lack of memory. Now none of them work, but there are no errors in console, except these... wait, I'll also post the ones Was gives for reference

t8ja commented 1 year ago

It's exactly the same like your VIDEO node.

`Error occurred when executing GroundingDinoSAMSegment (segment anything):

cannot unpack non-iterable NoneType object

File "/mnt/starT/AI/ComfyUI/execution.py", line 152, in recursive_execute output_data, output_ui = get_output_data(obj, input_data_all) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/starT/AI/ComfyUI/execution.py", line 82, in get_output_data return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/starT/AI/ComfyUI/execution.py", line 75, in map_node_over_list results.append(getattr(obj, func)(**slice_dict(input_data_all, i))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/starT/AI/ComfyUI/custom_nodes/comfyui_segment_anything/node.py", line 305, in main (images, masks) = sam_segment( ^^^^^^^^^^^^^^^`

t8ja commented 1 year ago

WAS works when I remove GroundingDINO from the loop. Now I'll try yours, tho that should've occurred to me earlier...

Nuked88 commented 1 year ago

No, no, your nodes never worked properly (but produced no ERRORS). I only got them to output 1 image at a time (including the preview video consisting of 1 frame).

WAS worked most of the time, until VRAM started getting low so sometimes it wouldn't render a few starting frames and then it would continue till the end, or fail due to lack of memory. Now none of them work, but there are no errors in console, except these... wait, I'll also post the ones Was gives for reference

without any context of your workflow is pretty impossible for me tell you the reason behind that, could be just your cpu that is slow (since my node is not using gpu except maybe the frameinterpolator) or just the fact that you are passing too much frames at the time without baching them. Just watch the task manager to see how much ram is used (not VRAM), if is too much then is possible that GPU is using your RAM because VRAM is full causing everything to run really slow. That situation could be caused by batch too big, no batch at all, or even frames too big in size (width X height) Let me know!

t8ja commented 1 year ago

Your nodes also work without DINO in the loop. The Save NODE even has the preview of the video, tho the LOADER is still blank.

Nuked88 commented 1 year ago

what do you mean "the loader still blank"?

t8ja commented 1 year ago

The preview on the LoadVideo node, I saw that you were worried about that aspect in the main UI repo.

Nuked88 commented 1 year ago

Oh this is probably a problem with the encoding of the original video, try to put the final video outputted by the SaveVideo in the LoadVideo

Nuked88 commented 1 year ago

no lol i mean the mp4 created by the SaveVideo! Drag it in the load video and see if the preview of the LoadVideo is working for that video

t8ja commented 1 year ago

Oh this is probably a problem with the encoding of the original video, try to put the final video outputted by the SaveVideo in the LoadVideo

You're right, now it shows the preview

Nuked88 commented 1 year ago

perfect!

t8ja commented 1 year ago

Appears all this could be due to the "threshold" setting on the DINO, 0.3 or above creates the error, but there are still weird things happening, like your nodes appearing without the tickable options to save, or without the button to select the input folder, but then they suddenly appear once I pick them anew from the list. All in all they work. EDIT: Spoke too soon

t8ja commented 1 year ago

`got prompt Temporary folder /mnt/starT/AI/ComfyUI/temp/n-suite/frames has been emptied. The video has 30 frames per second. Temporary folder /mnt/starT/AI/ComfyUI/output/n-suite/frames_out has been emptied. The video has been reduced to 7 frames per second. 70 frames have been extracted from the video and saved in /mnt/starT/AI/ComfyUI/output/n-suite/frames_out MoviePy - Writing audio in /mnt/starT/AI/ComfyUI/temp/audio.mp3 MoviePy - Done.
LR SCALES: [0.08589934592000005, 0.10737418240000006, 0.13421772800000006, 0.1677721600000001, 0.20971520000000007, 0.2621440000000001, 0.3276800000000001, 0.4096000000000001, 0.5120000000000001, 0.6400000000000001, 0.8, 1.0]

final text_encoder_type: bert-base-uncased Temporary folder /mnt/starT/AI/ComfyUI/temp/n-suite/frames has been emptied. Prompt executed in 85.09 seconds ` No output, still upredictable, using the same workflow.json ofc
t8ja commented 1 year ago

Your LoadVIdeo

frames=70 + batch=70 + threshold=0.5 = no output + error frames=70 + batch=70 + threshold=0.3 = no output frames=70 + batch=2 + threshold=0.3 = no output frames=70 + batch=1 + threshold=0.3 = output frames=70 + batch=1 + threshold=0.35 = no output + error

t8ja commented 1 year ago

Input image size might be the actual culprit, width of 512 doesn't error at least till 0.6

Nuked88 commented 1 year ago

man i think you haven't really grasped the concept of "batch" xD Having 70 frames and set a batch size of 70 is the equivalent of not using batches.

Using batches is a way for you to split the work of your GPU divinding some big work in "subworks"

If you have 70 frames i will start, for example, with a batch size of 10 so that you can split the work in 7 (70/10) "subworks"