search

kijai / ComfyUI-segment-anything-2

ComfyUI nodes to use segment-anything-2
Apache License 2.0
541 stars 33 forks source link

Clips w/ people moving in and out of frame #33

Open jacq819 opened 1 month ago

jacq819 commented 1 month ago

Hi! I have a video where people are coming in and out of frame and I want to generate a mask for all persons found in the clip. My prompt in the workflow using Florence2 is "person". It appears that the moment the first person found exits the frame, the segmentation stops for the remainder of the clip, even though more people appear eventually. I can see that Florence2 can detect people beyond that point for the duration of the clip, but the segmentation drops off and rather than saving out a mask clip that is the same frame length as my source video, its much shorter. I've tried "people" and "humans" for alt prompts but same result. I also set the frame rate correctly as well as the "num_frames" to my clip length. Any suggestions?

kijai commented 1 month ago

Yeah the video segmenter doesn't support that yet, it would need adding new points in the moment the subject re-appears. Alternative for now could be using the single_image segmenter in batch mode, that does each frame individually, but that requires pretty much perfect tracking.