To calc mAP , how many objects should use in py_metrics_ops.motion_metrics() ?

[An-Pan]

I have a motion prediction model, And i run all the val dataset and submit the result to https://waymo.com/open/challenges/2022/motion-prediction/. The result show the mAP in motion prediction taks is around 0.4376. But when I use the 'py_metrics_ops.motion_metrics' interface I got confused. If i use all the val data(190k) in motion_metrics() at once I got poor result, mAP=0.07. If i only use 100 objects in motion_metrics() the result mAP=0.56. If i use 1000 objects in motion_metrics() the result mAP=0.446 So , my question is how to use the motion_metrics() to do the same thing as 2022-motion_prediction-challenges in my local env.

My config file is : config_text = """ track_steps_per_second: 10 prediction_steps_per_second: 2 track_history_samples: 10 track_future_samples: 80 step_configurations { measurement_step: 5 lateral_miss_threshold: 1.0 longitudinal_miss_threshold: 2.0 } step_configurations { measurement_step: 9 lateral_miss_threshold: 1.8 longitudinal_miss_threshold: 3.6 } step_configurations { measurement_step: 15 lateral_miss_threshold: 3.0 longitudinal_miss_threshold: 6.0 } max_predictions: 6 speed_scale_lower: 0.5 speed_scale_upper: 1.0 speed_lower_bound: 1.4 speed_upper_bound: 11.0 """

[An-Pan]

Today, i add some log in motion_metrics.cc , and i find when only put 100 object in 'py_metrics_ops.motion_metrics' interface, there are only a few samples used per bucket to calc AP, and the ap result is 1. I think this is the main reason for mAP result is very different when different num of objects input. So, I want to know , how many objects should put into 'py_metrics_ops.motion_metrics' interface, to achive the same performace as 2022 waymo motion-prediction challenge.

[An-Pan]

I have a motion prediction model, And i run all the val dataset and submit the result to https://waymo.com/open/challenges/2022/motion-prediction/. The result show the mAP in motion prediction taks is around 0.4376. But when I use the 'py_metrics_ops.motion_metrics' interface I got confused. If i use all the val data(190k) in motion_metrics() at once I got poor result, mAP=0.07. If i only use 100 objects in motion_metrics() the result mAP=0.56. If i use 1000 objects in motion_metrics() the result mAP=0.446 So , my question is how to use the motion_metrics() to do the same thing as 2022-motion_prediction-challenges in my local env.

My config file is : config_text = """ track_steps_per_second: 10 prediction_steps_per_second: 2 track_history_samples: 10 track_future_samples: 80 step_configurations { measurement_step: 5 lateral_miss_threshold: 1.0 longitudinal_miss_threshold: 2.0 } step_configurations { measurement_step: 9 lateral_miss_threshold: 1.8 longitudinal_miss_threshold: 3.6 } step_configurations { measurement_step: 15 lateral_miss_threshold: 3.0 longitudinal_miss_threshold: 6.0 } max_predictions: 6 speed_scale_lower: 0.5 speed_scale_upper: 1.0 speed_lower_bound: 1.4 speed_upper_bound: 11.0 """

This is the result we get from 2022-waymo challenge

[scott-ettinger]

Hi. The metrics server code uses all of the tracks_to_predict objects in all of the scenarios in the validation set to compute the metrics. Which objects are you including?

[An-Pan]

The picture show the objects i used（all validation set）. The prediction data size is [1,192172,6,1,16,2], gt_data size is [1,192172,91,7].

The first dim is 1, that means i put all the val data as one scenario. Should I change the first dim to the real number of scenarios ? While, I read the metrics code , I don't think this is the reason of difference mAP result.

[scott-ettinger]

Sorry for the late reply. Could you try changing the first dimension to the number of scenarios?