search

XuyangBai / TransFusion

[PyTorch] Official implementation of CVPR2022 paper "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers". https://arxiv.org/abs/2203.11496
Apache License 2.0
619 stars 76 forks source link

Tracking results on NuScenes validation set #25

Open zhangpur opened 2 years ago

zhangpur commented 2 years ago

Thank you for sharing. Could you provide the tracking result file for the NuScenes validation set?

XuyangBai commented 2 years ago

Hi, sorry I am not able to share the tracking results on the val set as the model checkpoint is not allowed to sahre. You may train the model by yourself and then perform offline tracking following CenterPoint.

wzmsltw commented 2 years ago

@XuyangBai Hi Xuyang, using my trained Transfusion-L model (mAP 63.9 and NDS 68.9) and the CenterPoint nusc tracking codes, I get the tracking results on nusc validation set as: AMOTA 66.6 AMOTP 59.1. But it is far lower than reported AMOTA 70.5.

I wonder that if there is any point I should modify during using CenterPoint tracking codes? Thanks so much.

wzmsltw commented 2 years ago
Method det mAP det NDS track AMOTA track AMOTP
CenterPoint 59.5 67.4 66.8 57.3
Transfusion-L (re-produced version) 63.9 68.9 66.6 59.1
Transfusion-L (author version) 65.1 70.2 70.3 55.3

It seems my tracking results are not consistent with my detection results.

XuyangBai commented 2 years ago

Hi @wzmsltw I didn't change anything from CenterPoint's tracking method. Did you find the reason for the lower mAP & NDS of the detection part? I am suspecting that the problem is still from the detection part.

It seems my tracking results are not consistent with my detection results.

It could happen that your detection result is better but the tracking result is not, because 1) for tracking we use the velocity-based center distance as the matching cost, so the velocity prediction has a very large effect on the tracking part but it may have a smaller effect on detection NDS.

wzmsltw commented 2 years ago

I compare the metrics_summary.json of CenterPoint origin result and Transfusion result. And I find the main reason is that the AMOTA of transfusion pedestrain category is 20% lower than CenterPoint. Meanwhile, the detection mAP of transfusion pedestrian is a litter higher than CenterPoint. Thus I guess the main reason is the velocity of pedestrian. I will check the results carefully. Thanks for your reply!

wzmsltw commented 2 years ago

I find the main reason is that I did not filter detections with low score. This issue mainly affect the performance of Pedestrian. After filtering low score detections (using thres 0.01). AMOTA 69.1 is achieved.

XuyangBai commented 2 years ago

@wzmsltw Cool, glad to see you have solved it.