validation error KeyError

[rujiawang329]

Hi!

Because i only have 1 RTX 3090 so far,i use the python ./tools/train.py projects/configs/baselines/CAM-R50_img1600_128x128x10.py to train the model. However, I encountered the following error during verification.

2023-03-15 09:29:00,871 - mmdet - INFO - Epoch [1][28000/28130] lr: 2.000e-04, eta: 9 days, 6:50:28, time: 1.236, data_time: 0.019, memory: 12727, loss_depth: 1.0000, loss_voxel_ce_c_0: 1.0000, loss_voxel_sem_scal_c_0: 1.0000, loss_voxel_geo_scal_c_0: 1.0000, loss_voxel_lovasz_c_0: 1.0000, loss: 5.0000, grad_norm: 9.6413 2023-03-15 09:30:02,689 - mmdet - INFO - Epoch [1][28050/28130] lr: 2.000e-04, eta: 9 days, 6:49:23, time: 1.236, data_time: 0.018, memory: 12727, loss_depth: 1.0000, loss_voxel_ce_c_0: 1.0000, loss_voxel_sem_scal_c_0: 1.0000, loss_voxel_geo_scal_c_0: 1.0000, loss_voxel_lovasz_c_0: 1.0000, loss: 5.0000, grad_norm: 10.6449 2023-03-15 09:31:04,539 - mmdet - INFO - Epoch [1][28100/28130] lr: 2.000e-04, eta: 9 days, 6:48:18, time: 1.237, data_time: 0.021, memory: 12727, loss_depth: 1.0000, loss_voxel_ce_c_0: 1.0000, loss_voxel_sem_scal_c_0: 1.0000, loss_voxel_geo_scal_c_0: 1.0000, loss_voxel_lovasz_c_0: 1.0000, loss: 5.0000, grad_norm: 10.4283 2023-03-15 09:31:42,041 - mmdet - INFO - Saving checkpoint at 1 epochs [ ] 0/6019, elapsed: 0s, ETA:/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/torch/utils/checkpoint.py:25: UserWarning: None of the inputs have requires_grad=True. Gradients will be None warnings.warn("None of the inputs have requires_grad=True. Gradients will be None") Traceback (most recent call last): File "./tools/train.py", line 262, in main() File "./tools/train.py", line 251, in main custom_train_model( File "/media/re/2384a6b4-4dae-400d-ad72-9b7044491b55/data/OpenOccupancy-main/projects/occ_plugin/occupancy/apis/train.py", line 27, in custom_train_model custom_train_detector( File "/media/re/2384a6b4-4dae-400d-ad72-9b7044491b55/data/OpenOccupancy-main/projects/occ_plugin/occupancy/apis/mmdet_train.py", line 199, in custom_train_detector runner.run(data_loaders, cfg.workflow) File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run epoch_runner(data_loaders[i], **kwargs) File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 54, in train self.call_hook('after_train_epoch') File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmcv/runner/base_runner.py", line 307, in call_hook getattr(hook, fn_name)(self) File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmcv/runner/hooks/evaluation.py", line 267, in after_train_epoch self._do_evaluate(runner) File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmdet/core/evaluation/eval_hooks.py", line 17, in _do_evaluate results = single_gpu_test(runner.model, self.dataloader, show=False) File "/home/re/anaconda3/envs/open-mmlab/lib/python3.8/site-packages/mmdet/apis/test.py", line 59, in single_gpu_test if isinstance(result[0], tuple): KeyError: 0

[JeffWang987]

We have fixed the single GPU validation hooks, please pull the latest code. Also, you can resume your training by add --resume-from $CKPT

[rujiawang329]

Thanks!