Hichengdong
commented
1 month ago 运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py -b 1 --gpus 1 进行训练 运行python bevdepth/exps/nuscenes/mv/bev_depth_lss_r50_256x704_128x128_24e_ema.py --ckpt_path outputs/bev_depth_lss_r50_256x704_128x128_24e_ema/lightning_logs/version_2/21.pth -e -b 1 --gpus 1 进行测试
我使用mini数据集进行了训练，在第22个epoch的时候，程序卡死，然后我对最后的两个epoch进行了验证，结果map出奇的低。 ================epoch 20================= AP: 0.0003
mATE: 1.0000 mASE: 1.0000 mAOE: 1.0000 mAVE: 1.0000 mAAE: 1.0000 NDS: 0.0001 Eval time: 0.4s
Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.003 1.000 1.000 1.000 1.000 1.000 truck 0.000 1.000 1.000 1.000 1.000 1.000 bus 0.000 1.000 1.000 1.000 1.000 1.000 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.000 1.000 1.000 1.000 1.000 1.000 motorcycle 0.000 1.000 1.000 1.000 1.000 1.000 bicycle 0.000 1.000 1.000 1.000 1.000 1.000 traffic_cone 0.000 1.000 1.000 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan ================epoch 21================= AP: 0.0003
mATE: 1.0000 mASE: 1.0000 mAOE: 1.0000 mAVE: 1.0000 mAAE: 1.0000 NDS: 0.0002 Eval time: 0.4s
Per-class results: Object Class AP ATE ASE AOE AVE AAE car 0.003 1.000 1.000 1.000 1.000 1.000 truck 0.000 1.000 1.000 1.000 1.000 1.000 bus 0.000 1.000 1.000 1.000 1.000 1.000 trailer 0.000 1.000 1.000 1.000 1.000 1.000 construction_vehicle 0.000 1.000 1.000 1.000 1.000 1.000 pedestrian 0.000 1.000 1.000 1.000 1.000 1.000 motorcycle 0.000 1.000 1.000 1.000 1.000 1.000 bicycle 0.000 1.000 1.000 1.000 1.000 1.000 traffic_cone 0.000 1.000 1.000 nan nan nan barrier 0.000 1.000 1.000 1.000 nan nan Testing DataLoader 0: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 162/162 [00:29<00:00, 5.46it/s] TEST Profiler Report
| Action | Mean duration (s) | Num calls | Total time (s) | Percentage % |
| Total | - | 2823 | 87.467 | 100 % |
| run_test_evaluation | 31.038 | 1 | 31.038 | 35.486 | | [Strategy]SingleDeviceStrategy.test_step | 0.15896 | 162 | 25.751 | 29.441 | | [LightningModule]BEVDepthLightningModel.test_epoch_end | 2.6193 | 1 | 2.6193 | 2.9946 | | [EvaluationEpochLoop].test_dataloader_idx_0_next | 0.010395 | 162 | 1.6839 | 1.9252 | | [Strategy]SingleDeviceStrategy.batch_to_device | 0.003483 | 162 | 0.56424 | 0.64509 | | [LightningModule]BEVDepthLightningModel.test_dataloader | 0.099374 | 1 | 0.099374 | 0.11361 | | [Callback]TQDMProgressBar.on_test_batch_end | 0.00025195 | 162 | 0.040816 | 0.046664 | | [Callback]ModelSummary.on_test_batch_end | 9.2947e-06 | 162 | 0.0015057 | 0.0017215 | | [Callback]TQDMProgressBar.on_test_batch_start | 8.4548e-06 | 162 | 0.0013697 | 0.0015659 | | [LightningModule]BEVDepthLightningModel.on_test_model_train | 0.0010831 | 1 | 0.0010831 | 0.0012383 | | [LightningModule]BEVDepthLightningModel.on_test_model_eval | 0.00094745 | 1 | 0.00094745 | 0.0010832 | | [Callback]EMACallback.on_test_batch_start | 4.0783e-06 | 162 | 0.00066068 | 0.00075535 | | [Callback]TQDMProgressBar.on_test_start | 0.00061182 | 1 | 0.00061182 | 0.00069948 | | [Callback]EMACallback.on_test_batch_end | 2.4168e-06 | 162 | 0.00039152 | 0.00044762 | | [LightningModule]BEVDepthLightningModel.test_step_end | 1.8431e-06 | 162 | 0.00029859 | 0.00034137 | | [Callback]TQDMProgressBar.on_test_end | 0.00028655 | 1 | 0.00028655 | 0.00032761 | | [Callback]GradientAccumulationScheduler.on_test_batch_end | 1.4585e-06 | 162 | 0.00023627 | 0.00027012 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_end | 1.4072e-06 | 162 | 0.00022797 | 0.00026063 | | [Callback]ModelSummary.on_test_batch_start | 1.3618e-06 | 162 | 0.00022062 | 0.00025223 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_batch_start | 1.3274e-06 | 162 | 0.00021504 | 0.00024585 | | [Callback]GradientAccumulationScheduler.on_test_batch_start | 1.1475e-06 | 162 | 0.0001859 | 0.00021253 | | [LightningModule]BEVDepthLightningModel.on_test_batch_end | 1.0331e-06 | 162 | 0.00016737 | 0.00019135 | | [LightningModule]BEVDepthLightningModel.on_test_batch_start | 9.7408e-07 | 162 | 0.0001578 | 0.00018041 | | [Strategy]SingleDeviceStrategy.test_step_end | 8.8581e-07 | 162 | 0.0001435 | 0.00016406 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.setup | 8.6112e-05 | 1 | 8.6112e-05 | 9.845e-05 | | [Callback]TQDMProgressBar.on_test_epoch_start | 1.1989e-05 | 1 | 1.1989e-05 | 1.3707e-05 | | [Callback]EMACallback.on_test_epoch_end | 9.393e-06 | 1 | 9.393e-06 | 1.0739e-05 | | [Callback]ModelSummary.on_test_end | 8.157e-06 | 1 | 8.157e-06 | 9.3258e-06 | | [Callback]ModelSummary.on_test_start | 6.458e-06 | 1 | 6.458e-06 | 7.3833e-06 | | [Callback]TQDMProgressBar.setup | 5.747e-06 | 1 | 5.747e-06 | 6.5704e-06 | | [Callback]EMACallback.teardown | 5.065e-06 | 1 | 5.065e-06 | 5.7907e-06 | | [LightningModule]BEVDepthLightningModel.on_load_checkpoint | 4.902e-06 | 1 | 4.902e-06 | 5.6044e-06 | | [Callback]EMACallback.on_test_start | 4.671e-06 | 1 | 4.671e-06 | 5.3403e-06 | | [LightningModule]BEVDepthLightningModel.on_test_epoch_end | 4.038e-06 | 1 | 4.038e-06 | 4.6166e-06 | | [LightningModule]BEVDepthLightningModel.configure_callbacks | 3.997e-06 | 1 | 3.997e-06 | 4.5697e-06 | | [Callback]EMACallback.setup | 3.806e-06 | 1 | 3.806e-06 | 4.3513e-06 | | [LightningModule]BEVDepthLightningModel.configure_sharded_model | 3.26e-06 | 1 | 3.26e-06 | 3.7271e-06 | | [Callback]EMACallback.on_epoch_end | 3.089e-06 | 1 | 3.089e-06 | 3.5316e-06 | | [Callback]EMACallback.on_configure_sharded_model | 2.58e-06 | 1 | 2.58e-06 | 2.9497e-06 | | [Callback]EMACallback.on_epoch_start | 2.052e-06 | 1 | 2.052e-06 | 2.346e-06 | | [Callback]TQDMProgressBar.on_test_epoch_end | 2.044e-06 | 1 | 2.044e-06 | 2.3369e-06 | | [Callback]EMACallback.on_test_epoch_start | 2.032e-06 | 1 | 2.032e-06 | 2.3232e-06 | | [LightningModule]BEVDepthLightningModel.prepare_data | 1.966e-06 | 1 | 1.966e-06 | 2.2477e-06 | | [Callback]ModelSummary.on_test_epoch_start | 1.854e-06 | 1 | 1.854e-06 | 2.1196e-06 | | [Callback]EMACallback.on_before_accelerator_backend_setup | 1.802e-06 | 1 | 1.802e-06 | 2.0602e-06 | | [Callback]EMACallback.on_test_end | 1.636e-06 | 1 | 1.636e-06 | 1.8704e-06 | | [Callback]TQDMProgressBar.on_epoch_end | 1.597e-06 | 1 | 1.597e-06 | 1.8258e-06 | | [LightningModule]BEVDepthLightningModel.on_test_start | 1.552e-06 | 1 | 1.552e-06 | 1.7744e-06 | | [Callback]TQDMProgressBar.teardown | 1.542e-06 | 1 | 1.542e-06 | 1.7629e-06 | | [Callback]GradientAccumulationScheduler.on_test_start | 1.4e-06 | 1 | 1.4e-06 | 1.6006e-06 | | [Callback]GradientAccumulationScheduler.on_test_epoch_start | 1.349e-06 | 1 | 1.349e-06 | 1.5423e-06 | | [Callback]TQDMProgressBar.on_configure_sharded_model | 1.318e-06 | 1 | 1.318e-06 | 1.5068e-06 | | [Callback]ModelSummary.setup | 1.296e-06 | 1 | 1.296e-06 | 1.4817e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_start | 1.245e-06 | 1 | 1.245e-06 | 1.4234e-06 | | [Callback]ModelSummary.on_configure_sharded_model | 1.23e-06 | 1 | 1.23e-06 | 1.4062e-06 | | [Callback]TQDMProgressBar.on_before_accelerator_backend_setup | 1.215e-06 | 1 | 1.215e-06 | 1.3891e-06 | | [Callback]GradientAccumulationScheduler.setup | 1.214e-06 | 1 | 1.214e-06 | 1.3879e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.teardown | 1.163e-06 | 1 | 1.163e-06 | 1.3296e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_end | 1.148e-06 | 1 | 1.148e-06 | 1.3125e-06 | | [LightningModule]BEVDepthLightningModel.teardown | 1.147e-06 | 1 | 1.147e-06 | 1.3113e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_configure_sharded_model | 1.131e-06 | 1 | 1.131e-06 | 1.2931e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': None}.on_before_accelerator_backend_setup| 1.12e-06 | 1 | 1.12e-06 | 1.2805e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_end | 1.11e-06 | 1 | 1.11e-06 | 1.269e-06 | | [Callback]GradientAccumulationScheduler.on_test_end | 1.107e-06 | 1 | 1.107e-06 | 1.2656e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_start | 1.08e-06 | 1 | 1.08e-06 | 1.2347e-06 | | [Callback]GradientAccumulationScheduler.on_configure_sharded_model | 1.069e-06 | 1 | 1.069e-06 | 1.2222e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_test_epoch_start | 1.054e-06 | 1 | 1.054e-06 | 1.205e-06 | | [Callback]GradientAccumulationScheduler.on_before_accelerator_backend_setup | 1.037e-06 | 1 | 1.037e-06 | 1.1856e-06 | | [Callback]ModelSummary.on_test_epoch_end | 1.032e-06 | 1 | 1.032e-06 | 1.1799e-06 | | [Callback]ModelCheckpoint{'monitor': None, 'mode': 'min', 'every_n_train_steps': 0, 'every_n_epochs': 1, 'train_time_interval': None, 'save_on_train_epoch_end': True}.on_epoch_end | 1.026e-06 | 1 | 1.026e-06 | 1.173e-06 | | [Callback]ModelSummary.teardown | 1.023e-06 | 1 | 1.023e-06 | 1.1696e-06 | | [Callback]ModelSummary.on_before_accelerator_backend_setup | 9.86e-07 | 1 | 9.86e-07 | 1.1273e-06 | | [Callback]TQDMProgressBar.on_epoch_start | 9.76e-07 | 1 | 9.76e-07 | 1.1158e-06 | | [LightningModule]BEVDepthLightningModel.setup | 9.64e-07 | 1 | 9.64e-07 | 1.1021e-06 | | [Callback]GradientAccumulationScheduler.on_epoch_start | 9.51e-07 | 1 | 9.51e-07 | 1.0873e-06 | | [Callback]GradientAccumulationScheduler.teardown | 9.42e-07 | 1 | 9.42e-07 | 1.077e-06 | | [LightningModule]BEVDepthLightningModel.on_test_end | 8.99e-07 | 1 | 8.99e-07 | 1.0278e-06 | | [Callback]ModelSummary.on_epoch_end | 8.9e-07 | 1 | 8.9e-07 | 1.0175e-06 | | [Callback]ModelSummary.on_epoch_start | 8.88e-07 | 1 | 8.88e-07 | 1.0152e-06 | | [Callback]GradientAccumulationScheduler.on_test_epoch_end | 8.81e-07 | 1 | 8.81e-07 | 1.0072e-06 | | [Callback]GradientAccumulationScheduler.on_epoch_end | 8.39e-07 | 1 | 8.39e-07 | 9.5922e-07 | | [LightningModule]BEVDepthLightningModel.on_test_epoch_start | 7.55e-07 | 1 | 7.55e-07 | 8.6318e-07 | | [Strategy]SingleDeviceStrategy.on_test_start | 7.51e-07 | 1 | 7.51e-07 | 8.586e-07 | | [Strategy]SingleDeviceStrategy.on_test_end | 7.13e-07 | 1 | 7.13e-07 | 8.1516e-07 | | [LightningModule]BEVDepthLightningModel.on_epoch_start | 6.39e-07 | 1 | 6.39e-07 | 7.3056e-07 | | [LightningModule]BEVDepthLightningModel.on_epoch_end | 5.36e-07 | 1 | 5.36e-07 | 6.128e-07 |