SAM2 video failed to track the object for other frames when annotating the first frame

[jianchaoci]

Search before asking

• [X] I have searched the X-AnyLabeling Docs and issues and found no similar questions.

Question

Hi, Thank you for the amazing work. But I found a problem when I use this tool. I was using SAM2 video to annotate the object in a video, which is a simple cup. I gave prompt (positive point) on the cup, then it was segmented successfully. When I clicking 'lable all images', I found no more cups in the rest frames was annotated successfully.

Additional

I would expecte to see the cup in the rest images can be annotated successfully.

[CVHub520]

Hi, @jianchaoci

Thank you for your feedback and for using our tool. I understand that you're experiencing an issue with object tracking in SAM2 video when annotating frames.

To help diagnose and resolve this problem, could you please provide more detailed information? Specifically, can you check the terminal output logs and look for any error messages or warnings related to the annotation process?

Additionally, if possible, please share the following details:

1. The version of X-AnyLabeling you are using.
2. Any specific settings or configurations you have applied.

This information will be very helpful in troubleshooting the issue. Thank you!

Best regards, X-AnyLabeling Maintainer

[PairZhu]

The same problem to me. X-AnyLabeling: the latest commit from github log: No related log.(label_widget:run_all_images:5919 - Start running all images...)

[PairZhu]

@CVHub520 Could you provide the source of the dataset for the video demonstration so that I can check if it is an issue with my dataset?

[PairZhu]

My problem has been solved. The operation steps of the software are relatively strict. I can reason normally according to the following steps: Open X-AnyLabeling-> Load sam2video model-> Click Reset Tracker -> Create tags with sam2video-> Click Auto run all images at once I'm not sure if some of the steps are unnecessary or not in a strict order, but I can reason properly with this one. I don't know if it works for you. @jianchaoci

[jianchaoci]

My problem has been solved. The operation steps of the software are relatively strict. I can reason normally according to the following steps: Open X-AnyLabeling-> Load sam2video model-> Click Reset Tracker -> Create tags with sam2video-> Click Auto run all images at once I'm not sure if some of the steps are unnecessary or not in a strict order, but I can reason properly with this one. I don't know if it works for you. @jianchaoci

I have solved this problem caused by another reason. I checked the log and found there was actually a problem of SAM2 installation. Show as following when 'annotating all images':

I solved it by run 'python setup.py build_ext --inplace' in segment-anything-2 folder. Then reopen the labeling app.

[PairZhu]

My problem has been solved. The operation steps of the software are relatively strict. I can reason normally according to the following steps: Open X-AnyLabeling-> Load sam2video model-> Click Reset Tracker -> Create tags with sam2video-> Click Auto run all images at once I'm not sure if some of the steps are unnecessary or not in a strict order, but I can reason properly with this one. I don't know if it works for you. @jianchaoci

I have solved this problem caused by another reason. I checked the log and found there was actually a problem of SAM2 installation. Show as following when 'annotating all images':

I solved it by run 'python setup.py build_ext --inplace' in segment-anything-2 folder. Then reopen the labeling app.

You can find the solution to this problem in other issues：https://github.com/CVHub520/segment-anything-2/blob/main/INSTALL.md

You can see the build error using the commands below:

pip uninstall -y SAM-2 && \
rm -f ./sam2/*.so && \
SAM2_BUILD_ALLOW_ERRORS=0 pip install -v -e ".[demo]"

[yevhen-k]

I'd like to share similar issue.

Assume I have a video with different object to track. It has frames k, k+1, k+2, k+3,...

If new object appears on the frame, for example, k+2, I have to:

• cancel "Auto run all images at once" task
• go to the frame k+2
• reset tracker
• press CTRL-J
• add new bbox/points to track new object
• start "Auto run all images at once" task from the current k+2 frame As the result only bboxes for new object are labeled on k+3 and subsequent frames. All bboxes for other objects are gone.

Alternative case. If new object appears on the frame, for example, k+2, I have to:

• cancel "Auto run all images at once" task
• go to the frame k+2
• don't reset tracker
• press CTRL-J
• add new bbox/points to track new object
• start "Auto run all images at once" task from the current k+2 frame As the result only bboxes for old object are labeled on k+3 and subsequent frames. All bboxes for new objects are absent.

Wanted behavior if new object appears in the video:

• I'm able to cancel automated labeling
• I'm able to add new labels for newly appeared objects
• I'm able to start automated labeling form where it was stopped to add new labels
• labels for new object persist along with labels for old ones

At the moment I have to fully relabel frame to make object tracker see all labels after its reset.

[CVHub520]

Hey @yevhen-k, thanks for sharing this issue in detail!

I have provided a workflow that I hope may help solve this problem.

When a new object appears on frame k+2, here's a streamlined approach:

1. Cancel the "Auto run all images at once" task
2. Navigate to frame k+2
3. Reset the tracker
4. Press CTRL+Delete to clear any existing annotated objects on frame k+2
5. Press CTRL+J to switch edit mode and add bounding boxes/points for ALL objects you want to track (both old and new ones)
6. Restart the "Auto run all images at once" task from frame k+2

This approach should maintain tracking for both existing and new objects, avoiding the issue where labels for either old or new objects disappear. The key difference is that we're re-initializing tracking for all objects together, rather than trying to handle new objects separately.

Let me know if you try this workflow and if it helps with your use case!