Unable to reproduce the results of pp_multihead with nuScenes NDS:57.80

[kelsieyuan]

I use use 10 sweeps and the default settings with fixed seeds to train the nuscenes dataset for 20 epochs. I trained from scratch for several times, and the NDS could not reach to 58.23. can you share the training log or the best params?

Hardware:8*A100 FP32 pytorch Docker image: nvcr.io/nvidia/pytorch:20.07-py3

I run the following code: python -m pcdet.datasets.nuscenes.nuscenes_dataset --func create_nuscenes_infos \ --cfg_file tools/cfgs/dataset_configs/nuscenes_dataset.yaml \ --version v1.0-trainval sh scripts/dist_train.sh 8 --cfg_file cfgs/cbgs_pp_multihead.yaml --fix_random_seed

The results are as follows: 2022-04-09 16:38:16,129 INFO **Start evaluation workspace/cfgs/cbgs_pp_multihead(default)** 2022-04-09 16:38:16,130 INFO Loading NuScenes dataset 2022-04-09 16:38:16,598 INFO Total samples for NuScenes dataset: 6019 2022-04-09 16:38:16,601 INFO ==> Loading parameters from checkpoint /workspace/OpenPCDet/output/workspace/cfgs/cbgs_pp_multihead/default/ckpt/checkpoint_epoch_20.pth to CPU 2022-04-09 16:38:16,653 INFO ==> Checkpoint trained from version: pcdet+0.5.2+274c90c 2022-04-09 16:38:16,680 INFO ==> Done (loaded 421/421) 2022-04-09 16:38:16,684 INFO * EPOCH 20 EVALUATION *** 2022-04-09 16:39:43,660 INFO * Performance of EPOCH 20 *** 2022-04-09 16:39:43,660 INFO Generate label finished(sec_per_example: 0.0144 second). 2022-04-09 16:39:43,661 INFO recall_roi_0.3: 0.000000 2022-04-09 16:39:43,661 INFO recall_rcnn_0.3: 0.717701 2022-04-09 16:39:43,661 INFO recall_roi_0.5: 0.000000 2022-04-09 16:39:43,661 INFO recall_rcnn_0.5: 0.505958 2022-04-09 16:39:43,661 INFO recall_roi_0.7: 0.000000 2022-04-09 16:39:43,661 INFO recall_rcnn_0.7: 0.228869 2022-04-09 16:39:43,664 INFO Average predicted number of objects(6019 samples): 97.325 2022-04-09 16:42:03,741 INFO The predictions of NuScenes have been saved to /workspace/OpenPCDet/output/workspace/cfgs/cbgs_pp_multihead/default/eval/eval_with_train/epoch_20/val/final_result/data/results_nusc.json 2022-04-09 16:44:20,968 INFO ----------------Nuscene detection_cvpr_2019 results----------------- car error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.19, 0.15, 0.13, 0.28, 0.21 | 70.46, 81.17, 84.55, 86.45 | mean AP: 0.8065517421488804 truck error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.36, 0.19, 0.14, 0.23, 0.24 | 31.86, 47.42, 56.40, 60.32 | mean AP: 0.49000381674527216 construction_vehicle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.80, 0.46, 0.77, 0.12, 0.33 | 0.46, 6.95, 20.65, 26.64 | mean AP: 0.13674186930239868 bus error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.34, 0.18, 0.05, 0.44, 0.28 | 38.60, 59.03, 72.81, 75.79 | mean AP: 0.6155620440044398 trailer error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.58, 0.20, 0.41, 0.19, 0.16 | 8.66, 27.80, 47.86, 56.26 | mean AP: 0.3514400817021165 barrier error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.35, 0.28, 0.08, nan, nan | 30.25, 46.79, 54.04, 56.64 | mean AP: 0.469301953382638 motorcycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.23, 0.25, 0.39, 0.60, 0.27 | 26.28, 31.22, 31.79, 32.04 | mean AP: 0.30333830576103804 bicycle error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.19, 0.27, 0.53, 0.27, 0.04 | 7.10, 7.40, 7.41, 7.61 | mean AP: 0.07376244290071149 pedestrian error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.17, 0.28, 0.40, 0.26, 0.09 | 68.07, 70.30, 72.00, 74.12 | mean AP: 0.7112087715556576 traffic_cone error@trans, scale, orient, vel, attr | AP@0.5, 1.0, 2.0, 4.0 0.18, 0.34, nan, nan, nan | 41.51, 43.32, 45.82, 50.40 | mean AP: 0.45261172499063107 --------------average performance------------- trans_err: 0.3391 scale_err: 0.2606 orient_err: 0.3232 vel_err: 0.2991 attr_err: 0.2036 mAP: 0.4411 NDS: 0.5780

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.