Question about reproducing results using provided config file.

linleyzoe commented 1 year ago

Hi, thanks for your inspiring work! Following vic3d/late-fusion-pointcloud/pointpillars/README.md, the reproduced results using trainval_config_v.py and trainval_config_v.py are not as good as those using your provided checkpoints (inf-model & veh-model).

the reproduced results

car 3d IoU threshold 0.30, Average Precision = 60.87
car 3d IoU threshold 0.50, Average Precision = 49.24
car 3d IoU threshold 0.70, Average Precision = 29.07
car bev IoU threshold 0.30, Average Precision = 64.03
car bev IoU threshold 0.50, Average Precision = 54.57
car bev IoU threshold 0.70, Average Precision = 44.08
Average Communication Cost = 927.07 Bytes

results from provided checkpoints

car 3d IoU threshold 0.30, Average Precision = 63.40
car 3d IoU threshold 0.50, Average Precision = 53.36
car 3d IoU threshold 0.70, Average Precision = 37.28
car bev IoU threshold 0.30, Average Precision = 65.26
car bev IoU threshold 0.50, Average Precision = 59.16
car bev IoU threshold 0.70, Average Precision = 50.53
Average Communication Cost = 897.99 Bytes

How to reproduce the results from your provided checkpoints? Any advice would be greatly appreciated!

hhjj233 commented 1 year ago

In fact, I think the writer didn't train the network carefully. I choose VIC-Async-2 as the dataset, in which infrastructure has 2 frames delay than vehicle-side for each pair of pointcloud data. And my fusion result is much better than the result reported in the paper. Here is my result

car 3d IoU threshold 0.30, Average Precision = 59.46
car 3d IoU threshold 0.50, Average Precision = 48.30
car 3d IoU threshold 0.70, Average Precision = 28.85
car bev IoU threshold 0.30, Average Precision = 62.70
car bev IoU threshold 0.50, Average Precision = 53.49
car bev IoU threshold 0.70, Average Precision = 42.79
Average Communication Cost = 879.31 Bytes

hhjj233 commented 1 year ago

And I wonder which setting did you choose in the experiment, in which inf-model and veh-model provided by author, the result is not the same as any results listed in the paper.

haibao-yu commented 1 year ago

Yes, all experiment results of point cloud is much better than the paper report, because we keep improving the experiment details and code.

haibao-yu commented 1 year ago

Hi, thanks for your inspiring work! Following vic3d/late-fusion-pointcloud/pointpillars/README.md, the reproduced results using trainval_config_v.py and trainval_config_v.py are not as good as those using your provided checkpoints (inf-model & veh-model).

the reproduced results
car 3d IoU threshold 0.30, Average Precision = 60.87
car 3d IoU threshold 0.50, Average Precision = 49.24
car 3d IoU threshold 0.70, Average Precision = 29.07
car bev IoU threshold 0.30, Average Precision = 64.03
car bev IoU threshold 0.50, Average Precision = 54.57
car bev IoU threshold 0.70, Average Precision = 44.08
Average Communication Cost = 927.07 Bytes
results from provided checkpoints
car 3d IoU threshold 0.30, Average Precision = 63.40
car 3d IoU threshold 0.50, Average Precision = 53.36
car 3d IoU threshold 0.70, Average Precision = 37.28
car bev IoU threshold 0.30, Average Precision = 65.26
car bev IoU threshold 0.50, Average Precision = 59.16
car bev IoU threshold 0.70, Average Precision = 50.53
Average Communication Cost = 897.99 Bytes
How to reproduce the results from your provided checkpoints? Any advice would be greatly appreciated!

Did you prepare the data as the instructions?

AIR-THU / DAIR-V2X