search

hustvl / MapTR

[ICLR'23 Spotlight & IJCV'24] MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction
MIT License
1.08k stars 163 forks source link

Can not reproduce the performance of maptr with bevpool #88

Open JingweiZhang12 opened 1 year ago

JingweiZhang12 commented 1 year ago

Hi，the training of maptr with bevpool has some errors that some parameters are not used to compute loss, such as position_encoding. Althrough I have fixed them, the performance of maptr with bevpool still has a large gap with the reported one and the training is crashed caused by
File "MapTR/projects/mmdet3d_plugin/maptr/assigners/maptr_assigner.py", line 182, in assign 1950 【2023-08-28 13:03:46】 matched_row_inds, matched_col_inds = linear_sum_assignment(cost) 1951 【2023-08-28 13:03:46】ValueError: matrix contains invalid numeric entries The following shows the perfomance of the checkpoint of the 2nd epoch before the model training crash.
image I noticed that NuscMap_chamfer/boundary_AP in the 2nd epoch is 0.1950 in the log you provided.

LegendBC commented 1 year ago

Emmm, it is strange. Have you prepared the pretrained resnet50 backbone? If yes , and can you successfully reproduced the experiments with BEVFormer and GKT?

JingweiZhang12 commented 1 year ago

Emmm, it is strange. Have you prepared the pretrained resnet50 backbone? If yes , and can you successfully reproduced the experiments with BEVFormer and GKT?

I used the pretrained resnet50 backbone and I have successfully reproduce the experiments with BEVFormer and GKT. If possible, you could run the training using the config of MapTR with bevpool in this repo and I guess the same problems will arise.

LegendBC commented 1 year ago

I rerun the bevpool experiment with the released code,which is waiting in a queue. Once I have the intermediate results, I will inform in this thread.

LegendBC commented 1 year ago

My intermediate result seems fine. I guess that the problem may lie in the mmdet3d op compilation. image

LegendBC commented 1 year ago

@JingweiZhang12 Which type of GPU do you use for training? It may be attributed to the mmdet3d compilation flag.

hht1996ok commented 10 months ago

@LegendBC Could you provide the version of the environment? In my environment(A100*8), the effect using maptr with LSS is very poor. my environment: mmdet3d is same as https://github.com/mit-han-lab/bevfusion/tree/db75150717a9462cb60241e36ba28d65f6908607 mmdet==2.20.0 mmcv-full==1.4.0 mmsegmentation==0.14.1

hht1996ok commented 10 months ago

By the way, the bev features extracted by my model can be used normally in detection tasks. Strangely, when using MapTR head, there is a significant difference in performance compared to the original metric