What is the main metrics for ranking in E2E forecasting leaderboard?

foolhard commented 1 year ago

Hello,

I have some questions about the leaderboard metrics:

What is the main metrics for ranking in E2E forecasting leaderboard? Forecasting mAP, ADE, or FDE?
Why is the local metrics generated by test_forecast.py different from the leaderboard server?

Thanks.

neeharperi commented 1 year ago

The main metric for ranking is mAP_F
Are you using ROI pruning locally (by passing in a path to dataset_dir)? We use this in the test evaluation.

foolhard commented 1 year ago

@neeharperi I already input the dataset_dir to evaluation as follows: res = evaluate(forecasts, labels, config.top_k, config.ego_distance_threshold, paths["dataset_dir"]) As the sample code doesn't calculate mAP-F, I calculate mAP-F according to your paper： mAP-F = (mAP_static + mAP_linear + mAP_non-linear) / 3

Is this correct?

neeharperi commented 1 year ago

Yes, this is correct. Can you share a link to your predictions and we can investigate this further?

foolhard commented 1 year ago

I put my forecast_predictions.pkl in google drive.

I also submit the forecast file to the leaderboard but cannot get outputs for a long time. I may update when server results come out.
My forecaste test results in local PC are mAP_F 0.271596.

Is this result correct?

Please find details below:


{'linear': {'ARTICULATED_BUS': {'ADE': 14.553, 'FDE': 22.233, 'mAP_F': 0.126},
            'BICYCLE': {'ADE': 1.049, 'FDE': 1.666, 'mAP_F': 0.481},
            'BICYCLIST': {'ADE': 0.885, 'FDE': 1.423, 'mAP_F': 0.667},
            'BOLLARD': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'BOX_TRUCK': {'ADE': 2.602, 'FDE': 4.11, 'mAP_F': 0.283},
            'BUS': {'ADE': 1.836, 'FDE': 2.709, 'mAP_F': 0.357},
            'CONSTRUCTION_BARREL': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'CONSTRUCTION_CONE': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'DOG': {'ADE': 0.535, 'FDE': 0.741, 'mAP_F': 0.288},
            'LARGE_VEHICLE': {'ADE': 2.841, 'FDE': 3.682, 'mAP_F': 0.024},
            'MESSAGE_BOARD_TRAILER': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'MOBILE_PEDESTRIAN_CROSSING_SIGN': {'ADE': nan,
                                                'FDE': nan,
                                                'mAP_F': nan},
            'MOTORCYCLE': {'ADE': 1.387, 'FDE': 2.03, 'mAP_F': 0.224},
            'MOTORCYCLIST': {'ADE': 1.362, 'FDE': 2.35, 'mAP_F': 0.314},
            'PEDESTRIAN': {'ADE': 0.695, 'FDE': 1.063, 'mAP_F': 0.665},
            'REGULAR_VEHICLE': {'ADE': 1.549, 'FDE': 2.453, 'mAP_F': 0.48},
            'SCHOOL_BUS': {'ADE': 2.29, 'FDE': 3.872, 'mAP_F': 0.358},
            'SIGN': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'STOP_SIGN': {'ADE': 6.114, 'FDE': 8.334, 'mAP_F': 0.072},
            'STROLLER': {'ADE': 0.606, 'FDE': 1.004, 'mAP_F': 0.564},
            'TRUCK': {'ADE': 3.141, 'FDE': 4.663, 'mAP_F': 0.177},
            'TRUCK_CAB': {'ADE': 1.481, 'FDE': 1.847, 'mAP_F': 0.199},
            'VEHICULAR_TRAILER': {'ADE': 2.405, 'FDE': 3.106, 'mAP_F': 0.226},
            'WHEELCHAIR': {'ADE': 50.0, 'FDE': 50.0, 'mAP_F': 0.062},
            'WHEELED_DEVICE': {'ADE': 0.788, 'FDE': 1.081, 'mAP_F': 0.364},
            'WHEELED_RIDER': {'ADE': 1.346, 'FDE': 1.801, 'mAP_F': 0.163}},
 'non-linear': {'ARTICULATED_BUS': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'BICYCLE': {'ADE': 2.289, 'FDE': 4.38, 'mAP_F': 0.013},
                'BICYCLIST': {'ADE': 2.294, 'FDE': 4.565, 'mAP_F': 0.012},
                'BOLLARD': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'BOX_TRUCK': {'ADE': 4.52, 'FDE': 8.177, 'mAP_F': 0.098},
                'BUS': {'ADE': 2.69, 'FDE': 4.988, 'mAP_F': 0.157},
                'CONSTRUCTION_BARREL': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'CONSTRUCTION_CONE': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'DOG': {'ADE': 1.138, 'FDE': 1.505, 'mAP_F': 0.045},
                'LARGE_VEHICLE': {'ADE': 1.868, 'FDE': 3.403, 'mAP_F': 0.024},
                'MESSAGE_BOARD_TRAILER': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'MOBILE_PEDESTRIAN_CROSSING_SIGN': {'ADE': nan,
                                                    'FDE': nan,
                                                    'mAP_F': nan},
                'MOTORCYCLE': {'ADE': 3.296, 'FDE': 6.046, 'mAP_F': 0.073},
                'MOTORCYCLIST': {'ADE': 2.866, 'FDE': 4.876, 'mAP_F': 0.219},
                'PEDESTRIAN': {'ADE': 1.379, 'FDE': 2.453, 'mAP_F': 0.082},
                'REGULAR_VEHICLE': {'ADE': 3.081, 'FDE': 5.977, 'mAP_F': 0.113},
                'SCHOOL_BUS': {'ADE': 2.915, 'FDE': 6.104, 'mAP_F': 0.084},
                'SIGN': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'STOP_SIGN': {'ADE': 2.147, 'FDE': 2.149, 'mAP_F': 0.013},
                'STROLLER': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'TRUCK': {'ADE': 2.542, 'FDE': 4.521, 'mAP_F': 0.132},
                'TRUCK_CAB': {'ADE': 4.258, 'FDE': 7.315, 'mAP_F': 0.057},
                'VEHICULAR_TRAILER': {'ADE': 6.581,
                                      'FDE': 10.441,
                                      'mAP_F': 0.035},
                'WHEELCHAIR': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
                'WHEELED_DEVICE': {'ADE': 2.582, 'FDE': 3.593, 'mAP_F': 0.087},
                'WHEELED_RIDER': {'ADE': 2.457, 'FDE': 4.784, 'mAP_F': 0.023}},
 'static': {'ARTICULATED_BUS': {'ADE': 0.751, 'FDE': 0.909, 'mAP_F': 0.286},
            'BICYCLE': {'ADE': 0.343, 'FDE': 0.386, 'mAP_F': 0.657},
            'BICYCLIST': {'ADE': 0.671, 'FDE': 1.21, 'mAP_F': 0.172},
            'BOLLARD': {'ADE': 0.178, 'FDE': 0.221, 'mAP_F': 0.77},
            'BOX_TRUCK': {'ADE': 0.502, 'FDE': 0.59, 'mAP_F': 0.573},
            'BUS': {'ADE': 0.594, 'FDE': 0.912, 'mAP_F': 0.474},
            'CONSTRUCTION_BARREL': {'ADE': 0.128, 'FDE': 0.167, 'mAP_F': 0.906},
            'CONSTRUCTION_CONE': {'ADE': 0.14, 'FDE': 0.187, 'mAP_F': 0.744},
            'DOG': {'ADE': 0.882, 'FDE': 1.077, 'mAP_F': 0.061},
            'LARGE_VEHICLE': {'ADE': 0.525, 'FDE': 0.642, 'mAP_F': 0.14},
            'MESSAGE_BOARD_TRAILER': {'ADE': nan, 'FDE': nan, 'mAP_F': nan},
            'MOBILE_PEDESTRIAN_CROSSING_SIGN': {'ADE': 0.29,
                                                'FDE': 0.405,
                                                'mAP_F': 0.757},
            'MOTORCYCLE': {'ADE': 0.329, 'FDE': 0.385, 'mAP_F': 0.627},
            'MOTORCYCLIST': {'ADE': 1.48, 'FDE': 2.386, 'mAP_F': 0.325},
            'PEDESTRIAN': {'ADE': 0.434, 'FDE': 0.549, 'mAP_F': 0.633},
            'REGULAR_VEHICLE': {'ADE': 0.304, 'FDE': 0.377, 'mAP_F': 0.852},
            'SCHOOL_BUS': {'ADE': 0.766, 'FDE': 1.153, 'mAP_F': 0.454},
            'SIGN': {'ADE': 0.24, 'FDE': 0.272, 'mAP_F': 0.331},
            'STOP_SIGN': {'ADE': 0.113, 'FDE': 0.148, 'mAP_F': 0.761},
            'STROLLER': {'ADE': 0.399, 'FDE': 0.461, 'mAP_F': 0.044},
            'TRUCK': {'ADE': 0.59, 'FDE': 0.735, 'mAP_F': 0.176},
            'TRUCK_CAB': {'ADE': 0.661, 'FDE': 0.791, 'mAP_F': 0.183},
            'VEHICULAR_TRAILER': {'ADE': 0.51, 'FDE': 0.585, 'mAP_F': 0.461},
            'WHEELCHAIR': {'ADE': 0.159, 'FDE': 0.151, 'mAP_F': 0.004},
            'WHEELED_DEVICE': {'ADE': 0.403, 'FDE': 0.432, 'mAP_F': 0.497},
            'WHEELED_RIDER': {'ADE': 0.084, 'FDE': 0.084, 'mAP_F': 0.001}}}
--------------------------------------------------------------------------------
static:
mAP_F    0.43556
ADE      0.45904
FDE      0.60860
linear:
mAP_F    0.30470
ADE      4.87325
FDE      6.00840
non_linear:
mAP_F    0.074529
ADE      2.876647
FDE      5.016294
dtype: float64]
--------------------------------------------------------------------------------
Average forecasting scores:
mAP_F    0.271596
ADE      2.736312
FDE      3.877765

foolhard commented 1 year ago

I submitted a pkl to leaderboard server of Forecasting Validation Set, but no results for 5 days. The status always keeps 'Submitted'. What happened to the server?

neeharperi commented 1 year ago

We reached out to the EvalAI team a few days ago and haven't heard back yet. I suspect there is an issue on their end. I will keep you updated on this thread when we have a resolution.

foolhard commented 1 year ago

Any update about the evalai server? When will the server be ready? Besides, when will AV2 challenges end? I don't see the deadline of competition in WAD2023 website.

neeharperi commented 1 year ago

@Redrew found the issue and is working to remedy it. I expect the server should be back live by Monday. The competition will end on June 11th.

neeharperi commented 1 year ago

@foolhard We've updated the EvalAI system, your submissions should process normally now

neeharperi / LT3D

What is the main metrics for ranking in E2E forecasting leaderboard? #5