Potential issues with nuscenes tracking evaluation metrics

[lixiaoyu2000]

Hi, I recently discovered a potentially unreasonable problem with nuscenes tracking metric:

• Problem: Each category FP, FN, IDS, Recall, etc. in the evaluation metrics_summary.json file are all indicators under the optimal MOTA. This may prevent the final evaluation result from showing the true recall and accuracy performance of the tracker. :open_mouth:
• Case: When I was developing the new tracker, after checking out metrics_details.json, I found that it could work at a higher recall (which baseline tracker cannot) for some categories, thus achieving MOTA value at that recall. However, when I looked at the metrics_summary.json, I found that FN did not drop as I expected. After checking out the evaluation script, I understood the specific meaning of FN. A rising problem is that it might be misleading for the reviewer/developer to present the FN indicator in the article/leaderboard. :worried:
• Suggestion: Perhaps it would be better to provide a metric that can demonstrate the maximum recall rate that the tracker can achieve. :eyes:

If I misunderstood something, please let me know. :smiley:

[whyekit-motional]

@lixiaoyu2000 in the context of tracking for autonomous driving, we are concerned with more than just the lowest number of false negatives (which is what max. recall would indicate), so it does not seem practical to measure the max. recall achievable by different methods (at least for the nuScenes tracking benchmark)

[lixiaoyu2000]

Thank you for your reply! I think not providing max recall may obscure the real factor of AMOTA improvement, because AMOTA will assign 0 to unreachable recall.:smiley:

[whyekit-motional]

@lixiaoyu2000 I don't think there are any widely-used tracking metrics that cover the max recall