search

open-mmlab / mmdetection3d

OpenMMLab's next-generation platform for general 3D object detection.
https://mmdetection3d.readthedocs.io/en/latest/
Apache License 2.0
5.31k stars 1.54k forks source link

TypeError: class `RepeatDataset` in mmengine/dataset/dataset_wrapper.py: class `KittiDataset` in mmdet3d/datasets/kitti_dataset.py: The annotations loaded from annotation file should be a dict, but got <class 'list'>! #2507

Closed faziii0 closed 1 year ago

faziii0 commented 1 year ago

python tools/train.py configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py 05/04 18:01:39 - mmengine - INFO -

System environment: sys.platform: linux Python: 3.8.16 (default, Mar 2 2023, 03:21:46) [GCC 11.2.0] CUDA available: True numpy_random_seed: 1144527329 GPU 0: NVIDIA GeForce RTX 3080 Ti CUDA_HOME: /home/fazal/anaconda3/envs/mmd NVCC: Cuda compilation tools, release 11.6, V11.6.124 GCC: gcc (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0 PyTorch: 1.13.1 PyTorch compiling details: PyTorch built with:

Runtime environment: cudnn_benchmark: False mp_cfg: {'mp_start_method': 'fork', 'opencv_num_threads': 0} dist_cfg: {'backend': 'nccl'} seed: None Distributed launcher: none Distributed training: False GPU number: 1

05/04 18:01:40 - mmengine - INFO - Config: voxel_size = [0.16, 0.16, 4] model = dict( type='VoxelNet', data_preprocessor=dict( type='Det3DDataPreprocessor', voxel=True, voxel_layer=dict( max_num_points=32, point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1], voxel_size=[0.16, 0.16, 4], max_voxels=(16000, 40000))), voxel_encoder=dict( type='PillarFeatureNet', in_channels=4, feat_channels=[64], with_distance=False, voxel_size=[0.16, 0.16, 4], point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]), middle_encoder=dict( type='PointPillarsScatter', in_channels=64, output_shape=[496, 432]), backbone=dict( type='SECOND', in_channels=64, layer_nums=[3, 5, 5], layer_strides=[2, 2, 2], out_channels=[64, 128, 256]), neck=dict( type='SECONDFPN', in_channels=[64, 128, 256], upsample_strides=[1, 2, 4], out_channels=[128, 128, 128]), bbox_head=dict( type='Anchor3DHead', num_classes=3, in_channels=384, feat_channels=384, use_direction_classifier=True, assign_per_class=True, anchor_generator=dict( type='AlignedAnchor3DRangeGenerator', ranges=[[0, -39.68, -0.6, 69.12, 39.68, -0.6], [0, -39.68, -0.6, 69.12, 39.68, -0.6], [0, -39.68, -1.78, 69.12, 39.68, -1.78]], sizes=[[0.8, 0.6, 1.73], [1.76, 0.6, 1.73], [3.9, 1.6, 1.56]], rotations=[0, 1.57], reshape_out=False), diff_rad_by_sin=True, bbox_coder=dict(type='DeltaXYZWLHRBBoxCoder'), loss_cls=dict( type='mmdet.FocalLoss', use_sigmoid=True, gamma=2.0, alpha=0.25, loss_weight=1.0), loss_bbox=dict( type='mmdet.SmoothL1Loss', beta=0.1111111111111111, loss_weight=2.0), loss_dir=dict( type='mmdet.CrossEntropyLoss', use_sigmoid=False, loss_weight=0.2)), train_cfg=dict( assigner=[ dict( type='Max3DIoUAssigner', iou_calculator=dict(type='mmdet3d.BboxOverlapsNearest3D'), pos_iou_thr=0.5, neg_iou_thr=0.35, min_pos_iou=0.35, ignore_iof_thr=-1), dict( type='Max3DIoUAssigner', iou_calculator=dict(type='mmdet3d.BboxOverlapsNearest3D'), pos_iou_thr=0.5, neg_iou_thr=0.35, min_pos_iou=0.35, ignore_iof_thr=-1), dict( type='Max3DIoUAssigner', iou_calculator=dict(type='mmdet3d.BboxOverlapsNearest3D'), pos_iou_thr=0.6, neg_iou_thr=0.45, min_pos_iou=0.45, ignore_iof_thr=-1) ], allowed_border=0, pos_weight=-1, debug=False), test_cfg=dict( use_rotate_nms=True, nms_across_levels=False, nms_thr=0.01, score_thr=0.1, min_bbox_size=0, nms_pre=100, max_num=50)) dataset_type = 'KittiDataset' data_root = 'data/kitti/' class_names = ['Pedestrian', 'Cyclist', 'Car'] point_cloud_range = [0, -39.68, -3, 69.12, 39.68, 1] input_modality = dict(use_lidar=True, use_camera=False) metainfo = dict(classes=['Pedestrian', 'Cyclist', 'Car']) backend_args = None db_sampler = dict( data_root='data/kitti/', info_path='data/kitti/kitti_dbinfos_train.pkl', rate=1.0, prepare=dict( filter_by_difficulty=[-1], filter_by_min_points=dict(Car=5, Pedestrian=5, Cyclist=5)), classes=['Pedestrian', 'Cyclist', 'Car'], sample_groups=dict(Car=15, Pedestrian=15, Cyclist=15), points_loader=dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), backend_args=None) train_pipeline = [ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict(type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True), dict( type='ObjectSample', db_sampler=dict( data_root='data/kitti/', info_path='data/kitti/kitti_dbinfos_train.pkl', rate=1.0, prepare=dict( filter_by_difficulty=[-1], filter_by_min_points=dict(Car=5, Pedestrian=5, Cyclist=5)), classes=['Pedestrian', 'Cyclist', 'Car'], sample_groups=dict(Car=15, Pedestrian=15, Cyclist=15), points_loader=dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), backend_args=None), use_ground_plane=True), dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5), dict( type='GlobalRotScaleTrans', rot_range=[-0.78539816, 0.78539816], scale_ratio_range=[0.95, 1.05]), dict( type='PointsRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]), dict( type='ObjectRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]), dict(type='PointShuffle'), dict( type='Pack3DDetInputs', keys=['points', 'gt_labels_3d', 'gt_bboxes_3d']) ] test_pipeline = [ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict( type='MultiScaleFlipAug3D', img_scale=(1333, 800), pts_scale_ratio=1, flip=False, transforms=[ dict( type='GlobalRotScaleTrans', rot_range=[0, 0], scale_ratio_range=[1.0, 1.0], translation_std=[0, 0, 0]), dict(type='RandomFlip3D'), dict( type='PointsRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]) ]), dict(type='Pack3DDetInputs', keys=['points']) ] eval_pipeline = [ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict(type='Pack3DDetInputs', keys=['points']) ] train_dataloader = dict( batch_size=6, num_workers=4, persistent_workers=True, sampler=dict(type='DefaultSampler', shuffle=True), dataset=dict( type='RepeatDataset', times=2, dataset=dict( type='KittiDataset', data_root='data/kitti/', ann_file='kitti_infos_train.pkl', data_prefix=dict(pts='training/velodyne_reduced'), pipeline=[ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict( type='LoadAnnotations3D', with_bbox_3d=True, with_label_3d=True), dict( type='ObjectSample', db_sampler=dict( data_root='data/kitti/', info_path='data/kitti/kitti_dbinfos_train.pkl', rate=1.0, prepare=dict( filter_by_difficulty=[-1], filter_by_min_points=dict( Car=5, Pedestrian=5, Cyclist=5)), classes=['Pedestrian', 'Cyclist', 'Car'], sample_groups=dict(Car=15, Pedestrian=15, Cyclist=15), points_loader=dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), backend_args=None), use_ground_plane=True), dict(type='RandomFlip3D', flip_ratio_bev_horizontal=0.5), dict( type='GlobalRotScaleTrans', rot_range=[-0.78539816, 0.78539816], scale_ratio_range=[0.95, 1.05]), dict( type='PointsRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]), dict( type='ObjectRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]), dict(type='PointShuffle'), dict( type='Pack3DDetInputs', keys=['points', 'gt_labels_3d', 'gt_bboxes_3d']) ], modality=dict(use_lidar=True, use_camera=False), test_mode=False, metainfo=dict(classes=['Pedestrian', 'Cyclist', 'Car']), box_type_3d='LiDAR', backend_args=None))) val_dataloader = dict( batch_size=1, num_workers=1, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='KittiDataset', data_root='data/kitti/', data_prefix=dict(pts='training/velodyne_reduced'), ann_file='kitti_infos_val.pkl', pipeline=[ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict( type='MultiScaleFlipAug3D', img_scale=(1333, 800), pts_scale_ratio=1, flip=False, transforms=[ dict( type='GlobalRotScaleTrans', rot_range=[0, 0], scale_ratio_range=[1.0, 1.0], translation_std=[0, 0, 0]), dict(type='RandomFlip3D'), dict( type='PointsRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]) ]), dict(type='Pack3DDetInputs', keys=['points']) ], modality=dict(use_lidar=True, use_camera=False), test_mode=True, metainfo=dict(classes=['Pedestrian', 'Cyclist', 'Car']), box_type_3d='LiDAR', backend_args=None)) test_dataloader = dict( batch_size=1, num_workers=1, persistent_workers=True, drop_last=False, sampler=dict(type='DefaultSampler', shuffle=False), dataset=dict( type='KittiDataset', data_root='data/kitti/', data_prefix=dict(pts='training/velodyne_reduced'), ann_file='kitti_infos_val.pkl', pipeline=[ dict( type='LoadPointsFromFile', coord_type='LIDAR', load_dim=4, use_dim=4, backend_args=None), dict( type='MultiScaleFlipAug3D', img_scale=(1333, 800), pts_scale_ratio=1, flip=False, transforms=[ dict( type='GlobalRotScaleTrans', rot_range=[0, 0], scale_ratio_range=[1.0, 1.0], translation_std=[0, 0, 0]), dict(type='RandomFlip3D'), dict( type='PointsRangeFilter', point_cloud_range=[0, -39.68, -3, 69.12, 39.68, 1]) ]), dict(type='Pack3DDetInputs', keys=['points']) ], modality=dict(use_lidar=True, use_camera=False), test_mode=True, metainfo=dict(classes=['Pedestrian', 'Cyclist', 'Car']), box_type_3d='LiDAR', backend_args=None)) val_evaluator = dict( type='KittiMetric', ann_file='data/kitti/kitti_infos_val.pkl', metric='bbox', backend_args=None) test_evaluator = dict( type='KittiMetric', ann_file='data/kitti/kitti_infos_val.pkl', metric='bbox', backend_args=None) vis_backends = [dict(type='LocalVisBackend')] visualizer = dict( type='Det3DLocalVisualizer', vis_backends=[dict(type='LocalVisBackend')], name='visualizer') lr = 0.001 optim_wrapper = dict( type='OptimWrapper', optimizer=dict( type='AdamW', lr=0.001, betas=(0.95, 0.99), weight_decay=0.01), clip_grad=dict(max_norm=35, norm_type=2)) param_scheduler = [ dict( type='CosineAnnealingLR', T_max=32.0, eta_min=0.01, begin=0, end=32.0, by_epoch=True, convert_to_iter_based=True), dict( type='CosineAnnealingLR', T_max=48.0, eta_min=1.0000000000000001e-07, begin=32.0, end=80, by_epoch=True, convert_to_iter_based=True), dict( type='CosineAnnealingMomentum', T_max=32.0, eta_min=0.8947368421052632, begin=0, end=32.0, by_epoch=True, convert_to_iter_based=True), dict( type='CosineAnnealingMomentum', T_max=48.0, eta_min=1, begin=32.0, end=80, convert_to_iter_based=True) ] train_cfg = dict(by_epoch=True, max_epochs=80, val_interval=2) val_cfg = dict() test_cfg = dict() auto_scale_lr = dict(enable=False, base_batch_size=48) default_scope = 'mmdet3d' default_hooks = dict( timer=dict(type='IterTimerHook'), logger=dict(type='LoggerHook', interval=50), param_scheduler=dict(type='ParamSchedulerHook'), checkpoint=dict(type='CheckpointHook', interval=-1), sampler_seed=dict(type='DistSamplerSeedHook'), visualization=dict(type='Det3DVisualizationHook')) env_cfg = dict( cudnn_benchmark=False, mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0), dist_cfg=dict(backend='nccl')) log_processor = dict(type='LogProcessor', window_size=50, by_epoch=True) log_level = 'INFO' load_from = None resume = False epoch_num = 80 launcher = 'none' work_dir = './work_dirs/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class'

/home/fazal/Downloads/UQ/mmd3d/mmdetection3d/mmdet3d/models/dense_heads/anchor3d_head.py:92: UserWarning: dir_offset and dir_limit_offset will be depressed and be incorporated into box coder in the future warnings.warn( 05/04 18:01:42 - mmengine - INFO - Distributed training is not used, all SyncBatchNorm (SyncBN) layers in the model will be automatically reverted to BatchNormXd layers if they are used. 05/04 18:01:42 - mmengine - INFO - Hooks will be executed in the following order: before_run: (VERY_HIGH ) RuntimeInfoHook
(BELOW_NORMAL) LoggerHook

before_train: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(VERY_LOW ) CheckpointHook

before_train_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(NORMAL ) DistSamplerSeedHook

before_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook

after_train_iter: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train_epoch: (NORMAL ) IterTimerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

before_val_epoch: (NORMAL ) IterTimerHook

before_val_iter: (NORMAL ) IterTimerHook

after_val_iter: (NORMAL ) IterTimerHook
(NORMAL ) Det3DVisualizationHook
(BELOW_NORMAL) LoggerHook

after_val_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook
(LOW ) ParamSchedulerHook
(VERY_LOW ) CheckpointHook

after_train: (VERY_LOW ) CheckpointHook

before_test_epoch: (NORMAL ) IterTimerHook

before_test_iter: (NORMAL ) IterTimerHook

after_test_iter: (NORMAL ) IterTimerHook
(NORMAL ) Det3DVisualizationHook
(BELOW_NORMAL) LoggerHook

after_test_epoch: (VERY_HIGH ) RuntimeInfoHook
(NORMAL ) IterTimerHook
(BELOW_NORMAL) LoggerHook

after_run: (BELOW_NORMAL) LoggerHook

05/04 18:01:43 - mmengine - INFO - load 14357 Car database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 2207 Pedestrian database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 734 Cyclist database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 1297 Van database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 56 Person_sitting database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 488 Truck database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 224 Tram database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 337 Misc database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - After filter database: 05/04 18:01:43 - mmengine - INFO - load 10520 Car database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 2066 Pedestrian database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 580 Cyclist database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 826 Van database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 53 Person_sitting database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 321 Truck database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 199 Tram database infos in DataBaseSampler 05/04 18:01:43 - mmengine - INFO - load 259 Misc database infos in DataBaseSampler Traceback (most recent call last): File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 122, in build_from_cfg obj = obj_cls(**args) # type: ignore File "/home/fazal/Downloads/UQ/mmd3d/mmdetection3d/mmdet3d/datasets/kitti_dataset.py", line 77, in init super().init( File "/home/fazal/Downloads/UQ/mmd3d/mmdetection3d/mmdet3d/datasets/det3d_dataset.py", line 129, in init super().init( File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 250, in init self.full_init() File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 301, in full_init self.data_list = self.load_data_list() File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/dataset/base_dataset.py", line 440, in load_data_list raise TypeError(f'The annotations loaded from annotation file ' TypeError: The annotations loaded from annotation file should be a dict, but got <class 'list'>!

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 122, in build_from_cfg obj = obj_cls(*args) # type: ignore File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/dataset/dataset_wrapper.py", line 211, in init self.dataset = DATASETS.build(dataset) File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/registry.py", line 548, in build return self.build_func(cfg, args, **kwargs, registry=self) File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 144, in build_from_cfg raise type(e)( TypeError: class KittiDataset in mmdet3d/datasets/kitti_dataset.py: The annotations loaded from annotation file should be a dict, but got <class 'list'>!

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "tools/train.py", line 135, in main() File "tools/train.py", line 131, in main runner.train() File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1687, in train self._train_loop = self.build_train_loop( File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1486, in build_train_loop loop = EpochBasedTrainLoop( File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/runner/loops.py", line 44, in init super().init(runner, dataloader) File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/runner/base_loop.py", line 26, in init self.dataloader = runner.build_dataloader( File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1346, in build_dataloader dataset = DATASETS.build(dataset_cfg) File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/registry.py", line 548, in build return self.build_func(cfg, *args, **kwargs, registry=self) File "/home/fazal/anaconda3/envs/mmd/lib/python3.8/site-packages/mmengine/registry/build_functions.py", line 144, in build_from_cfg raise type(e)( TypeError: class RepeatDataset in mmengine/dataset/dataset_wrapper.py: class KittiDataset in mmdet3d/datasets/kitti_dataset.py: The annotations loaded from annotation file should be a dict, but got <class 'list'>!

Xiangxu-0103 commented 1 year ago

You may need to use this script to update your info.

faziii0 commented 1 year ago

$ python tools/dataset_converters/update_infos_to_v2.py --dataset kitti --pkl-path ./data/kitti/kitti_infos_train.pkl --out-dir ./kitti_v2/ ./data/kitti/kitti_infos_train.pkl will be modified. Reading from input file: ./data/kitti/kitti_infos_train.pkl. Start updating: [ ] 0/3712, elapsed: 0s, ETA:Traceback (most recent call last): File "tools/dataset_converters/update_infos_to_v2.py", line 1159, in update_pkl_infos( File "tools/dataset_converters/update_infos_to_v2.py", line 1138, in update_pkl_infos update_kitti_infos(pkl_path=pkl_path, out_dir=out_dir) File "tools/dataset_converters/update_infos_to_v2.py", line 413, in update_kitti_infos temp_data_info['sample_idx'] = ori_info_dict['image']['image_idx'] KeyError: 'image'

faziii0 commented 1 year ago

I am using the kitti dataset for traning.

mmdetection3d ├── mmdet3d ├── tools ├── configs ├── data │ ├── kitti │ │ ├── ImageSets │ │ ├── testing │ │ │ ├── calib │ │ │ ├── image_2 │ │ │ ├── velodyne │ │ ├── training │ │ │ ├── calib │ │ │ ├── image_2 │ │ │ ├── label_2 │ │ │ ├── velodyne

Xiangxu-0103 commented 1 year ago

It seems there are some problems with your original info (may lose the information about images). You can also use the annotation files we generated offline here: https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_train.pkl, https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_val.pkl, https://download.openmmlab.com/mmdetection3d/data/kitti/kitti_infos_test.pkl, to have a try.

More details can be found here: https://github.com/open-mmlab/mmdetection3d/blob/dev-1.x/docs/en/user_guides/dataset_prepare.md#kitti

faziii0 commented 1 year ago

Thanks Xiangxu. Training has been started but after few min it throws this error.

05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3600/3769] eta: 0:00:02 time: 0.0145 data_time: 0.0006 memory: 302
05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3650/3769] eta: 0:00:01 time: 0.0146 data_time: 0.0006 memory: 285
05/09 10:44:31 - mmengine - INFO - Epoch(val) [2][3700/3769] eta: 0:00:01 time: 0.0145 data_time: 0.0006 memory: 284
05/09 10:44:32 - mmengine - INFO - Epoch(val) [2][3750/3769] eta: 0:00:00 time: 0.0146 data_time: 0.0006 memory: 289

Converting 3D prediction to KITTI format [>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 1485.0 task/s, elapsed: 3s, ETA: 0s Result is saved to /tmp/tmpuk72cs2o/results/pred_instances_3d.pkl. Traceback (most recent call last): File "tools/train.py", line 135, in main() File "tools/train.py", line 131, in main runner.train() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1721, in train model = self.train_loop.run() # type: ignore File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 102, in run self.runner.val_loop.run() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 366, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate _results = metric.evaluate(size) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 205, in compute_metrics ap_dict = self.kitti_evaluate( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 244, in kitti_evaluate ap_result_str, apdict = kitti_eval( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 725, in kitti_eval mAP40_3d, mAP40_aos = do_eval(gt_annos, dt_annos, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 626, in do_eval ret = eval_class(gt_annos, dt_annos, current_classes, difficultys, 1, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 480, in eval_class rets = calculate_iou_partly(dt_annos, gt_annos, metric, num_parts) File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 384, in calculate_iou_partly overlap_part = bev_box_overlap(dt_boxes, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 118, in bev_box_overlap from .rotate_iou import rotate_iou_gpu_eval File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 283, in def rotate_iou_kernel_eval(N, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit disp.compile(argtypes) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 927, in compile kernel = _Kernel(self.py_func, argtypes, self.targetoptions) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, *kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 84, in init cres = compile_cuda(self.py_func, types.void, self.argtypes, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/compiler.py", line 230, in compile_cuda cres = compiler.compile_extra(typingctx=typingctx, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 742, in compile_extra return pipeline.compile_extra(func) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 460, in compile_extra return self._compile_bytecode() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 528, in _compile_bytecode return self._compile_core() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 507, in _compile_core raise e File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 494, in _compile_core pm.run(self.state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 368, in run raise patched_exception File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 356, in run self._runPass(idx, pass_inst, state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, **kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass mutated |= check(pss.run_pass, internal_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 273, in check mangled = func(compiler_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 110, in run_pass typemap, return_type, calltypes, errs = type_inference_stage( File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 88, in type_inference_stage errs = infer.propagate(raise_errors=raise_errors) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typeinfer.py", line 1086, in propagate raise errors[0] numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fd742e50490>. libNVVM cannot be found. Do conda install cudatoolkit: libnvvm.so: cannot open shared object file: No such file or directory During: resolving callee type: type(CUDADispatcher(<function rbbox_to_corners at 0x7fd7427a00d0>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (241)

Enable logging at debug level for details.

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 241: def inter(rbbox1, rbbox2):

rbbox_to_corners(corners1, rbbox1)
^

During: resolving callee type: type(CUDADispatcher(<function inter at 0x7fd7427a0280>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (269)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 269: def devRotateIoUEval(rbox1, rbox2, criterion=-1):

area2 = rbox2[2] * rbox2[3]
area_inter = inter(rbox1, rbox2)
^

During: resolving callee type: type(CUDADispatcher(<function devRotateIoUEval at 0x7fd7427a0430>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (332)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 332: def rotate_iou_kernel_eval(N,

            tx * K + i)
        dev_iou[offset] = devRotateIoUEval(block_qboxes[i * 5:i * 5 + 5],
Xiangxu-0103 commented 1 year ago

It seems some errors occurred with CUDA. You may need to check whether the CUDA has been installed correctly, or try to conda install cudatoolkit.

faziii0 commented 1 year ago

Thanks for your help. All solved. cheers

airainday commented 1 year ago

Thanks Xiangxu. Training has been started but after few min it throws this error.

05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3600/3769] eta: 0:00:02 time: 0.0145 data_time: 0.0006 memory: 302 05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3650/3769] eta: 0:00:01 time: 0.0146 data_time: 0.0006 memory: 285 05/09 10:44:31 - mmengine - INFO - Epoch(val) [2][3700/3769] eta: 0:00:01 time: 0.0145 data_time: 0.0006 memory: 284 05/09 10:44:32 - mmengine - INFO - Epoch(val) [2][3750/3769] eta: 0:00:00 time: 0.0146 data_time: 0.0006 memory: 289

Converting 3D prediction to KITTI format [>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 1485.0 task/s, elapsed: 3s, ETA: 0s Result is saved to /tmp/tmpuk72cs2o/results/pred_instances_3d.pkl. Traceback (most recent call last): File "tools/train.py", line 135, in main() File "tools/train.py", line 131, in main runner.train() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1721, in train model = self.train_loop.run() # type: ignore File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 102, in run self.runner.val_loop.run() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 366, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate _results = metric.evaluate(size) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 205, in compute_metrics ap_dict = self.kitti_evaluate( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 244, in kitti_evaluate ap_result_str, apdict = kitti_eval( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 725, in kitti_eval mAP40_3d, mAP40_aos = do_eval(gt_annos, dt_annos, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 626, in do_eval ret = eval_class(gt_annos, dt_annos, current_classes, difficultys, 1, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 480, in eval_class rets = calculate_iou_partly(dt_annos, gt_annos, metric, num_parts) File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 384, in calculate_iou_partly overlap_part = bev_box_overlap(dt_boxes, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 118, in bev_box_overlap from .rotate_iou import rotate_iou_gpu_eval File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 283, in def rotate_iou_kernel_eval(N, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit disp.compile(argtypes) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 927, in compile kernel = _Kernel(self.py_func, argtypes, self.targetoptions) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 84, in init* cres = compile_cuda(self.py_func, types.void, self.argtypes, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/compiler.py", line 230, in compile_cuda cres = compiler.compile_extra(typingctx=typingctx, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 742, in compile_extra return pipeline.compile_extra(func) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 460, in compile_extra return self._compile_bytecode() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 528, in _compile_bytecode return self._compile_core() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 507, in _compile_core raise e File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 494, in _compile_core pm.run(self.state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 368, in run raise patched_exception File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 356, in run self._runPass(idx, pass_inst, state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, **kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass mutated |= check(pss.run_pass, internal_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 273, in check mangled = func(compiler_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 110, in run_pass typemap, return_type, calltypes, errs = type_inference_stage( File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 88, in type_inference_stage errs = infer.propagate(raise_errors=raise_errors) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typeinfer.py", line 1086, in propagate raise errors[0] numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fd742e50490>. libNVVM cannot be found. Do conda install cudatoolkit: libnvvm.so: cannot open shared object file: No such file or directory During: resolving callee type: type(CUDADispatcher(<function rbbox_to_corners at 0x7fd7427a00d0>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (241)

Enable logging at debug level for details.

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 241: def inter(rbbox1, rbbox2):

rbbox_to_corners(corners1, rbbox1)
^

During: resolving callee type: type(CUDADispatcher(<function inter at 0x7fd7427a0280>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (269)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 269: def devRotateIoUEval(rbox1, rbox2, criterion=-1): area2 = rbox2[2] * rbox2[3] area_inter = inter(rbox1, rbox2) ^

During: resolving callee type: type(CUDADispatcher(<function devRotateIoUEval at 0x7fd7427a0430>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (332)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 332: def rotate_iou_kernel_eval(N, tx K + i) dev_iou[offset] = devRotateIoUEval(block_qboxes[i 5:i * 5 + 5],

Hello，How did you finally solve the problem

Assassincup commented 11 months ago

Thanks Xiangxu. Training has been started but after few min it throws this error. 05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3600/3769] eta: 0:00:02 time: 0.0145 data_time: 0.0006 memory: 302 05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3650/3769] eta: 0:00:01 time: 0.0146 data_time: 0.0006 memory: 285 05/09 10:44:31 - mmengine - INFO - Epoch(val) [2][3700/3769] eta: 0:00:01 time: 0.0145 data_time: 0.0006 memory: 284 05/09 10:44:32 - mmengine - INFO - Epoch(val) [2][3750/3769] eta: 0:00:00 time: 0.0146 data_time: 0.0006 memory: 289 Converting 3D prediction to KITTI format [>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 1485.0 task/s, elapsed: 3s, ETA: 0s Result is saved to /tmp/tmpuk72cs2o/results/pred_instances_3d.pkl. Traceback (most recent call last): File "tools/train.py", line 135, in main() File "tools/train.py", line 131, in main runner.train() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1721, in train model = self.train_loop.run() # type: ignore File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 102, in run self.runner.val_loop.run() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 366, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate _results = metric.evaluate(size) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 205, in compute_metrics ap_dict = self.kitti_evaluate( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 244, in kitti_evaluate ap_result_str, apdict = kitti_eval( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 725, in kitti_eval mAP40_3d, mAP40_aos = do_eval(gt_annos, dt_annos, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 626, in do_eval ret = eval_class(gt_annos, dt_annos, current_classes, difficultys, 1, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 480, in eval_class rets = calculate_iou_partly(dt_annos, gt_annos, metric, num_parts) File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 384, in calculate_iou_partly overlap_part = bev_box_overlap(dt_boxes, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 118, in bev_box_overlap from .rotate_iou import rotate_iou_gpu_eval File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 283, in def rotate_iou_kernel_eval(N, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit disp.compile(argtypes) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 927, in compile kernel = _Kernel(self.py_func, argtypes, self.targetoptions) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 84, in init* cres = compile_cuda(self.py_func, types.void, self.argtypes, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/compiler.py", line 230, in compile_cuda cres = compiler.compile_extra(typingctx=typingctx, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 742, in compile_extra return pipeline.compile_extra(func) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 460, in compile_extra return self._compile_bytecode() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 528, in _compile_bytecode return self._compile_core() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 507, in _compile_core raise e File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 494, in _compile_core pm.run(self.state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 368, in run raise patched_exception File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 356, in run self._runPass(idx, pass_inst, state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, **kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass mutated |= check(pss.run_pass, internal_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 273, in check mangled = func(compiler_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 110, in run_pass typemap, return_type, calltypes, errs = type_inference_stage( File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 88, in type_inference_stage errs = infer.propagate(raise_errors=raise_errors) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typeinfer.py", line 1086, in propagate raise errors[0] numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fd742e50490>. libNVVM cannot be found. Do conda install cudatoolkit: libnvvm.so: cannot open shared object file: No such file or directory During: resolving callee type: type(CUDADispatcher(<function rbbox_to_corners at 0x7fd7427a00d0>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (241) Enable logging at debug level for details. File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 241: def inter(rbbox1, rbbox2):

rbbox_to_corners(corners1, rbbox1)
^

During: resolving callee type: type(CUDADispatcher(<function inter at 0x7fd7427a0280>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (269) File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 269: def devRotateIoUEval(rbox1, rbox2, criterion=-1): area2 = rbox2[2] rbox2[3] area_inter = inter(rbox1, rbox2) ^ During: resolving callee type: type(CUDADispatcher(<function devRotateIoUEval at 0x7fd7427a0430>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (332) File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 332: def rotate_iou_kernel_eval(N, tx K + i) dev_iou[offset] = devRotateIoUEval(block_qboxes[i 5:i 5 + 5],

Hello，How did you finally solve the problem

Hello，did you solve the problem?I find this wrong too.Could you please tell me some information.

Assassincup commented 11 months ago

Thanks Xiangxu. Training has been started but after few min it throws this error.

05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3600/3769] eta: 0:00:02 time: 0.0145 data_time: 0.0006 memory: 302 05/09 10:44:30 - mmengine - INFO - Epoch(val) [2][3650/3769] eta: 0:00:01 time: 0.0146 data_time: 0.0006 memory: 285 05/09 10:44:31 - mmengine - INFO - Epoch(val) [2][3700/3769] eta: 0:00:01 time: 0.0145 data_time: 0.0006 memory: 284 05/09 10:44:32 - mmengine - INFO - Epoch(val) [2][3750/3769] eta: 0:00:00 time: 0.0146 data_time: 0.0006 memory: 289

Converting 3D prediction to KITTI format [>>>>>>>>>>>>>>>>>>>>>>>>>>] 3769/3769, 1485.0 task/s, elapsed: 3s, ETA: 0s Result is saved to /tmp/tmpuk72cs2o/results/pred_instances_3d.pkl. Traceback (most recent call last): File "tools/train.py", line 135, in main() File "tools/train.py", line 131, in main runner.train() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/runner.py", line 1721, in train model = self.train_loop.run() # type: ignore File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 102, in run self.runner.val_loop.run() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/runner/loops.py", line 366, in run metrics = self.evaluator.evaluate(len(self.dataloader.dataset)) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate _results = metric.evaluate(size) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate _metrics = self.compute_metrics(results) # type: ignore File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 205, in compute_metrics ap_dict = self.kitti_evaluate( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/metrics/kitti_metric.py", line 244, in kitti_evaluate ap_result_str, apdict = kitti_eval( File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 725, in kitti_eval mAP40_3d, mAP40_aos = do_eval(gt_annos, dt_annos, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 626, in do_eval ret = eval_class(gt_annos, dt_annos, current_classes, difficultys, 1, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 480, in eval_class rets = calculate_iou_partly(dt_annos, gt_annos, metric, num_parts) File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 384, in calculate_iou_partly overlap_part = bev_box_overlap(dt_boxes, File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/eval.py", line 118, in bev_box_overlap from .rotate_iou import rotate_iou_gpu_eval File "/home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 283, in def rotate_iou_kernel_eval(N, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/decorators.py", line 133, in _jit disp.compile(argtypes) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 927, in compile kernel = _Kernel(self.py_func, argtypes, self.targetoptions) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/dispatcher.py", line 84, in init* cres = compile_cuda(self.py_func, types.void, self.argtypes, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(args, kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/cuda/compiler.py", line 230, in compile_cuda cres = compiler.compile_extra(typingctx=typingctx, File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 742, in compile_extra return pipeline.compile_extra(func) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 460, in compile_extra return self._compile_bytecode() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 528, in _compile_bytecode return self._compile_core() File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 507, in _compile_core raise e File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler.py", line 494, in _compile_core pm.run(self.state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 368, in run raise patched_exception File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 356, in run self._runPass(idx, pass_inst, state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_lock.py", line 35, in _acquire_compile_lock return func(*args, **kwargs) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 311, in _runPass mutated |= check(pss.run_pass, internal_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/compiler_machinery.py", line 273, in check mangled = func(compiler_state) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 110, in run_pass typemap, return_type, calltypes, errs = type_inference_stage( File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typed_passes.py", line 88, in type_inference_stage errs = infer.propagate(raise_errors=raise_errors) File "/home/fazal/anaconda3/envs/mm1/lib/python3.8/site-packages/numba/core/typeinfer.py", line 1086, in propagate raise errors[0] numba.core.errors.TypingError: Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Failed in cuda mode pipeline (step: nopython frontend) Internal error at <numba.core.typeinfer.CallConstraint object at 0x7fd742e50490>. libNVVM cannot be found. Do conda install cudatoolkit: libnvvm.so: cannot open shared object file: No such file or directory During: resolving callee type: type(CUDADispatcher(<function rbbox_to_corners at 0x7fd7427a00d0>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (241)

Enable logging at debug level for details.

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 241: def inter(rbbox1, rbbox2):

rbbox_to_corners(corners1, rbbox1)
^

During: resolving callee type: type(CUDADispatcher(<function inter at 0x7fd7427a0280>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (269)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 269: def devRotateIoUEval(rbox1, rbox2, criterion=-1): area2 = rbox2[2] * rbox2[3] area_inter = inter(rbox1, rbox2) ^

During: resolving callee type: type(CUDADispatcher(<function devRotateIoUEval at 0x7fd7427a0430>)) During: typing of call at /home/fazal/Downloads/m3d/mmdetection3d/mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py (332)

File "mmdet3d/evaluation/functional/kitti_utils/rotate_iou.py", line 332: def rotate_iou_kernel_eval(N, tx K + i) dev_iou[offset] = devRotateIoUEval(block_qboxes[i 5:i * 5 + 5],

Hello,I find the same error,could you please tell me some information,I'll appreciate it.

faziii0 commented 11 months ago

most of them are cuda errors try to fix cuda and try different versions

Assassincup commented 11 months ago

most of them are cuda errors try to fix cuda and try different versions

Thanks for your rapid answer ,I just address the problem by "conda install cudatoolkit",but what confused me is that I have an anaconda environment with cuda and cudnn ,why should i install cudatoolkit again ,hahaha.

faziii0 commented 11 months ago

most of them are cuda errors try to fix cuda and try different versions

Thanks for your rapid answer ,I just address the problem by "conda install cudatoolkit",but what confused me is that I have an anaconda environment with cuda and cudnn ,why should i install cudatoolkit again ,hahaha.

They all work together as CUDA Toolkit: The CUDA Toolkit is a set of tools, libraries, and documentation provided by NVIDIA for GPU-accelerated computing. It includes the CUDA runtime, libraries like cuDNN, and development tools for building GPU-accelerated applications.