search

XuyangBai / TransFusion

[PyTorch] Official implementation of CVPR2022 paper "TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers". https://arxiv.org/abs/2203.11496
Apache License 2.0
619 stars 76 forks source link

loss_heatmap reduction abnormal #33

Open study1994 opened 2 years ago

study1994 commented 2 years ago

hillo，I trained the pandaset dataset with your code。The training went well with transfusion_pillar_L and training log see link：https://pan.baidu.com/s/1PFjYIedIrYY1qKw5SwUe1A?pwd=4cho Extraction code：4cho but bad with transfusion_pillar_LC and training log see link：https://pan.baidu.com/s/10en2ppjXWTYlhE_i4wyXqw?pwd=zxbb Extraction code：zxbb The performance of the model transfusion_pillar_LC trained on transfusion_pillar_L basis is degraded。What I'm confused about is that loss_heatmap fall first and then rise and the last matched_ious should be the highest。the MAP of the final test on data is lower than transfusion_pillar_L。 there is no problem in data validation，what do you think may be the wrong place and Can you give me some advice？

XuyangBai commented 2 years ago

Hi @study1994 Here are some suggestions:

  1. Try removing GlobalRotScaleTrans and RandomFlip3D when you train the LC model. These augmentations will bring some mismatch for the image-guided query initialization (although does not hurt the final performance on nuScenes) and I have already commented it out for TransFusionLC in the last commit. You can find some detailed discussion here
  2. Is your 2D backbone pre-trained on pandaset? If not you may first pre-train the 2D backbone on pandaset for detection or segmentation task so that the img feature can learn semantic on pandaset.