Error in training

[bhaskar-anand-iith]

I am getting following error after completing the 60 epochs,

File "train.py", line 118, in main() File "train.py", line 115, in main train_detector(model, datasets, cfg, distributed=distributed, validate=args.validate, logger=logger,) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/torchie/apis/train_sessd.py", line 323, in train_detector trainer.run(data_loaders, cfg.workflow, cfg.total_epochs, local_rank=cfg.local_rank) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 469, in run epoch_runner(data_loaders[2], *kwargs) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/torchie/trainer/trainer_sessd.py", line 407, in val resultdict, = self.data_loader.dataset.evaluation(predictions, output_dir=self.work_dir) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/kitti.py", line 157, in evaluation result_coco_dict = get_coco_eval_result(gt_annos, dt_annos, self._class_names, z_axis=z_axis, z_center=z_center) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/eval.py", line 739, in get_coco_eval_result z_center=z_center, File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/eval.py", line 437, in do_coco_style_eval min_overlaps[:, i, j] = np.linspace(overlap_ranges[:, i, j]) File "<__array_function__ internals>", line 6, in linspace File "/media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numpy/core/function_base.py", line 120, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer

Please help if you faced such issue..

[Vegeta2020]

@bhaskar-anand-iith Someone seems reported a similiar issue before, it's a bug about COCO eval API, so you may comment these code, as they are not useful nowadays.

[bhaskar-anand-iith]

Even while testing I am getting the same error. Please let me know which part is to be commented, since eval.py has fully coco eval API only.

(ssdenv) bhaskar@bhaskar:/media/bhaskar/Data_Disk/object_detection/SE-SSD/tools$ python test.py True Namespace(checkpoint='epoch_58.pth', config='../examples/second/configs/config.py', eval=None, eval_id=None, json_out='json_out.json', launcher='none', local_rank=0, out='out.pkl', show=False, tmpdir=None, txt_result=True, vis_id=None) [>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ] 3765/3768, 52.6 task/s, elapsed: 72s, ETA: 0s/media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (45) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (54) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (60) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (40) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (50) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (48) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (6) < 2 * SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/core/typed_passes.py:330: NumbaPerformanceWarning: The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.

To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.

File "../det3d/datasets/utils/eval.py", line 324: @numba.jit(nopython=True, parallel=True) def box3d_overlap_kernel(boxes, qboxes, rinc, criterion=-1, z_axis=1, z_center=1.0): ^

state.func_ir.loc)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (54) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (60) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (45) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (40) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (50) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (48) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) /media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numba/cuda/compiler.py:726: NumbaPerformanceWarning: Grid size (6) < 2 SM count (164) will likely result in GPU under utilization due to low occupancy. warn(NumbaPerformanceWarning(msg)) Traceback (most recent call last): File "test.py", line 269, in main() File "test.py", line 228, in main result_dict, detections = test(data_loader, model, save_dir=None, distributed=distributed) File "test.py", line 49, in test return dataset.evaluation(predictions, str(save_dir)) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/kitti.py", line 157, in evaluation result_coco_dict = get_coco_eval_result(gt_annos, dt_annos, self._class_names, z_axis=z_axis, z_center=z_center) File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/eval.py", line 739, in get_coco_eval_result z_center=z_center, File "/media/bhaskar/Data_Disk/object_detection/SE-SSD/det3d/datasets/kitti/eval.py", line 437, in do_coco_style_eval min_overlaps[:, i, j] = np.linspace(overlap_ranges[:, i, j]) File "<__array_function__ internals>", line 6, in linspace File "/media/bhaskar/Data_Disk/object_detection/ssdenv/lib/python3.7/site-packages/numpy/core/function_base.py", line 120, in linspace num = operator.index(num) TypeError: 'numpy.float64' object cannot be interpreted as an integer

[bhaskar-anand-iith]

Solved it..

Follow: https://github.com/Vegeta2020/SE-SSD/issues/81#issuecomment-1102136064