search

mrnabati / CenterFusion

CenterFusion: Center-based Radar and Camera Fusion for 3D Object Detection
MIT License
520 stars 141 forks source link

The val results does not correspond to the paper #57

Open gleeeee1 opened 2 years ago

gleeeee1 commented 2 years ago

I tested the network using the mini dataset val and weight centerfusion_e60.pth. And I got the following results. There is a gap between these results and those in paper. I think it might be because the dataset is too small. Do you use the val of the full dataset or the val from the mini dataset? How can the evaluation result be matched to the paper? `mAP: 0.3129 mATE: 0.6590 mASE: 0.4630 mAOE: 0.6008 mAVE: 0.7622 mAAE: 0.3084 NDS: 0.3771 Eval time: 7.8s

Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.542 0.418 0.159 0.120 0.152 0.073 truck 0.442 0.542 0.168 0.133 0.141 0.122 bus 0.549 0.484 0.081 0.079 1.400 0.090 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.468 0.561 0.262 0.491 0.512 0.118 motorcycle 0.318 0.689 0.342 1.167 0.061 0.016 bicycle 0.187 0.448 0.299 0.417 1.832 0.048 traffic_cone 0.624 0.447 0.319 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan`

here's the results from the paper: Method Dataset C R L NDS ↑ mAP ↑ mA TE mASE mAOE mA VE mAAE Ours (DLA) val X X 0.453 0.332 0.649 0.263 0.535 0.540 0.142 Method Dataset C R L Car Truck Bus Trailer Const. Pedest. Motor. Bicycle Traff. Barrier Ours (DLA) val X X 0.524 0.265 0.362 0.154 0.055 0.389 0.305 0.229 0.563 0.470 please give me some advice, thank you!

gleeeee1 commented 2 years ago

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

niranjanravilevelup commented 1 year ago

@gleeeee1 Hi, I have some trouble with using the mini-dataset. The issue is at https://github.com/mrnabati/CenterFusion/issues/62 . can you please help me with this? Thank you.

JYS997760473 commented 1 year ago

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

I was using the mini_test dataset. I think this is the mean problem. I have changed the dataset to val dataset.

Hi, do you evaluate on nuScenes validation dataset?