search

anDoer / PoseTrack21

MIT License
69 stars 10 forks source link

Incorrect computation of FPs in frames with zero GT annotations #10

Open krishnakanthnakka opened 2 years ago

krishnakanthnakka commented 2 years ago

Hi @anDoer,

Thank you for adapting the HOTA for keypoint tracking. I would like to bring to your notice about possible error in computing the False positives in images where there is no GT.

In line https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py#L127, the FP are added by variable num_tracker_joints .

However the variable num_tracker_joints is a vector of size (J,) containing number of all valid keypoints in entire video (which is huge value for long-sequence videos) and not just the detections in current frame.

Please see https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/datasets/posetrack.py#L349

In other words, the FP at time instant t is added by incorrect value and I observed HOTA significantly degrades by few frames (near the end of video) with no GT annotation

I think it should be modified to res['HOTA_FP'][a] += len(tracker_ids_t).

Furthermore, same analogy for FNs in the later lines with zero predictions. res['HOTA_FN'][a] += num_gt_joints to be modified to res['HOTA_FN'][a] += len(gt_ids_t ).

Please correct me if I'm wrong

anDoer commented 2 years ago

Hi @krishnakanthnakka thanks for your detailed report.

I will investigate this issue and provide a patch soon.

krishnakanthnakka commented 2 years ago

Hello @anDoer,

Thanks for the prompt response.

I want to inform you that the frames with no GT annotations labels, they are already filtered at https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/datasets/posetrack.py#L444 using islabelled field.

So, for the PoseTrack21, i think it won't enter this loop https://github.com/anDoer/PoseTrack21/blob/35bd7033ec4e1a352ae39b9522df5a683f83781b/eval/posetrack21/posetrack21/trackeval/metrics/hota_pose.py#L127

Please investigate this issue. I found this issue when visual tracked outputs and computed HOTA metrics are not consistent on a private sports dataset.