search

henry123-boy / SpaTracker

[CVPR 2024 Highlight] Official PyTorch implementation of SpatialTracker: Tracking Any 2D Pixels in 3D Space
Other
733 stars 25 forks source link

Evaluation code for TAP-VID benchmark #22

Open JennySeidenschwarz opened 5 months ago

JennySeidenschwarz commented 5 months ago

Hey everyone! :)

Congrats to your amazing work!! Do you have a ready-to-use file for evaluation on TAP-VID? Or do you have per-sequence performances?

Would be amazing!!

Thanks, Jenny

m43 commented 3 months ago

Hi Jenny, I get the following numbers for the DAVIS subset of TAP-Vid and the spaT_final.pth checkpoint shared by the authors:

tapvid_davis_avg_Jaccard    59.10832733877460
tapvid_davis_avg_delta      75.03060472325210
tapvid_davis_avg_OA     89.15536883348100

My per sequence results are here: tapvid_davis_metrics.csv

This does not match the aj=61.1,delta=76.3,oa=89.5 numbers reported in the paper, it is a bit worse than that. But I assume that this might be expected since it is not clear if the released checkpoint matches the full architecture described in the paper (e.g., with the ARAP embeddings). Maybe someone got similar results to the ones I have shared above?