search

alfredgu001324 / MapBEVPrediction

[ECCV 2024] Accelerating Online Mapping and Behavior Prediction via Direct BEV Feature Attention
https://arxiv.org/abs/2407.06683
Apache License 2.0
47 stars 8 forks source link

opposite conclusions #4

Open Guanyi-Liu opened 2 days ago

Guanyi-Liu commented 2 days ago

Hello, I have carefully read your paper and compared the DenseTNT metrics between the previous CVPR and the current ECCV experiments. I noticed that these two experiments seem to lead to almost completely opposite conclusions. Could you please explain why this happens and how to interpret this phenomenon? image image

alfredgu001324 commented 2 days ago

Thank you for your interest in our work!! Yes your finding is correct. My suspicion is that:

  1. Lack of hyperparameter sweep. Since in our ECCV work, we are now only considering the agents within the BEV perception range, it is theoretically a different experiment setting. But for convenience (and also time pressure), we reused most of the hyperparameters settings from our previous work. Theoretically, it is best to find a set of hyperparameters that show bev > uncertainty > baseline, but as soon as we found a setting such that bev shows a greater improvement %, we just decided to use that directly without any more finetuning.

  2. Another tricky part with DenseTNT is that it was originally designed to use centerlines to sample goals. But in our case, we are using either dividers or boundaries to sample goals (not all the online mapping methods output centerlines). This would actually cause the trajectories to deviate towards the boundaries. This is also why you can see a huge reduction in displacement errors when using MapTRv2-CL. So technically DenseTNT is not really compatible with online mapping in the first place. We chose DenseTNT mainly because it is a classical example of GNN-based model and it has a nice codebase.

Please let me know if you have further questions!