search

aharley / pips

Particle Video Revisited
MIT License
568 stars 51 forks source link

PointOdyssey Evaluation Protocol Details #37

Open m43 opened 1 year ago

m43 commented 1 year ago

Thank you for the stellar work and the new dataset! I'm trying to understand the exact evaluation protocol you used for evaluating models on PointOdyssey, specifically for the numbers reported in Table 2 of the PointOdyssey paper:

  1. Did you filter out trajectories that aren't visible at any given timestep?
  2. Were there any other trajectory filtering processes applied?
  3. How did you determine the query frame for individual trajectories? Perhaps as the first timestep a point became visible?
  4. When computing the average position accuracy (from TAP-Vid) and the Median Trajectory Error (MTE), did you exclude occluded points and points before the query frame?
  5. Is the survival rate adjusted for varying trajectory lengths within a video? For instance, some queries might first appear only toward the end of a video.

If this part of the code has been released, I could potentially delve into the details myself. Thanks for your assistance!

bhack commented 1 year ago

Very soon, this repo will also host the PIPs++ model from our ICCV 2023 paper

@aharley Any roadmap to release PIPs++?

aharley commented 1 year ago

Hi, sorry for the delay. I'm aiming for this Friday.

bhack commented 1 year ago

How it scales in term of training/inference resolution?