search

nutonomy / nuscenes-devkit

The devkit of the nuScenes dataset.
https://www.nuScenes.org
Other
2.19k stars 616 forks source link

Why is the default metric for the motp and amotp parameters in the "python-sdk/nuscenes/eval/tracking/configs/tracking_nips_2019.json" file set to 2.0? #1007

Closed hjfenghj closed 7 months ago

hjfenghj commented 7 months ago

Why is the default metric for the motp and amotp parameters in the "python-sdk/nuscenes/eval/tracking/configs/tracking_nips_2019.json" file set to 2.0? The higher the AMOTP metric, the better. If set in this way, will there be no problems when calculating the AMOTP values for each category image

whyekit-motional commented 7 months ago

@hjfenghj MOTP goes up to 1.0, so it shouldn't be an issue, I believe

hjfenghj commented 7 months ago

However, when calculating the AMOTP indicator for a single category, it is the mean of the MOTP corresponding to 40 threshold scores. When the recall corresponding to the threshold score is nan, the MOTP is 2.0. The MOTP calculated in this way will have a value greater than 1. How should AMOTP indicators be compared in this situation

---- Replied Message ---- | From | @.> | | Date | 11/16/2023 17:50 | | To | nutonomy/nuscenes-devkit @.> | | Cc | 冯浩俊 @.>, Mention @.> | | Subject | Re: [nutonomy/nuscenes-devkit] Why is the default metric for the motp and amotp parameters in the "python-sdk/nuscenes/eval/tracking/configs/tracking_nips_2019.json" file set to 2.0? (Issue #1007) |

@hjfenghj MOTP goes up to 1.0, so it shouldn't be an issue, I believe

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>

whyekit-motional commented 7 months ago

@hjfenghj when calculating AMOTP, this is how NaNs are handled: https://github.com/nutonomy/nuscenes-devkit/blob/9b165b1018a64623b65c17b64f3c9dd746040f36/python-sdk/nuscenes/eval/tracking/evaluate.py#L168-L171

hjfenghj commented 7 months ago

Yes, but the 170th line of code replaces the element corresponding to the nan value with 2.0. If there are 22 nan and 18 normal values. The final calculation of the average is [22 * 2.0+sum (18 normal values)]/40 image

hjfenghj commented 7 months ago

thanks，may I have understood the issue。beacuse the smaller the param is,the better .

nightrome commented 7 months ago

Yes, AMOTP is the localization error in meters. So a smaller AMOTP is better and thus it is okay to bound it at some (very bad) value.