search

IDEA-Research / TAPTR

[ECCV 2024] Official implementation of the paper "TAPTR: Tracking Any Point with Transformers as Detection"
https://taptr.github.io
Other
196 stars 12 forks source link

Real-Time Application of TAPTRv2 #2

Closed niharvanjara24 closed 3 months ago

niharvanjara24 commented 3 months ago

Hello Team,

Excellent work on TAPTRv2! I have a few questions regarding its suitability for real-time online applications:

  1. Can TAPTRv2 handle real-time tasks where frames are continuously received and query points are tracked on the fly?
  2. If so, are there any benchmarks available, particularly concerning tracking FPS (frames per second), for real-time applications?

Thank you in advance for your assistance.

LHY-HongyangLi commented 3 months ago

Hi @niharvanjara24 , Thank you for your attention!

  1. Sure as we have described in this issue: https://github.com/IDEA-Research/TAPTR/issues/4#issue-2441267398, TAPTR is able to be an online tracker after modifying the code for the preparation of the video features. But it may not be able to be real-time. Further engineering optimization is needed, such as replacing traditional attention in our self-attention and temporal-attention block with flash-attention or agent attention.
  2. As far as I have known, there is currently no such benchmark.
LHY-HongyangLi commented 3 months ago

Hi @niharvanjara24 , I'm sorry for missing this paper, which focuses on long-range point tracking: https://openaccess.thecvf.com/content/CVPR2024/papers/Balasingam_DriveTrack_A_Benchmark_for_Long-Range_Point_Tracking_in_Real-World_Videos_CVPR_2024_paper.pdf