LeongVan
commented
1 year ago It works now. I just simply stop and re-run it for several times, and then the program works.
Closed LeongVan closed 1 year ago
LeongVan
commented
1 year ago It works now. I just simply stop and re-run it for several times, and then the program works.
zhanglk9
commented
1 year ago 现在可以了。我只是简单地停止并重新运行它几次，然后程序就可以运行了。
I rerun many times, but it doesn't solve the problem
zhanglk9
commented
1 year ago I meet the same problem. Do you solve that now??
LeongVan
commented
1 year ago I meet the same problem. Do you solve that now??
Yes, just simply rerun it. Nothing else has changed.
lilkeker
commented
1 year ago I also encountered the same problem. I modified part of the code in LSS, and the debugging can be successful, but it will get stuck at this place. If I use the original code that has not been modified, everything is normal
lilkeker
commented
1 year ago I meet the same problem. Do you solve that now?? Hello, did you solve this problem？ I wonder if this is a bug
zhanglk9
commented
1 year ago I meet the same problem. Do you solve that now?? Hello, did you solve this problem？ I wonder if this is a bug
Today I use one GPU to test ,and I didn't meet this problem.But using 4 GPUs，it doesn't work.
LeongVan
commented
1 year ago I also encountered the same problem. I modified part of the code in LSS, and the debugging can be successful, but it will get stuck at this place. If I use the original code that has not been modified, everything is normal
I didn't changed the original code, but still met the problem. Yesterday, I trained on 4 GPUs, and when I checked out this morning, it stuck again.
lilkeker
commented
1 year ago I also encountered the same problem. I modified part of the code in LSS, and the debugging can be successful, but it will get stuck at this place. If I use the original code that has not been modified, everything is normal
I didn't changed the original code, but still met the problem. Yesterday, I trained on 4 GPUs, and when I checked out this morning, it stuck again. I'm wondering if it's a problem with the environment installation. This is the process of my environment installation. If I don't modify any code, there will be no problem running. You can refer to it
lilkeker
commented
1 year ago Hi, thanks for opening source program. I've got a question. I was running the evaluation using command
CUDA_VISIBLE_DEVICE=7 torchpack dist-run -np 4 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox(CUDA_VISIBLE_DEVICE=7 is used to control which GPU device is activated, '7' means I merely used the 8th one). But I got the following situation: the program seems to stuck without any process running. I've been waiting for half an hour and no process was going on. Is it normal or should I waiting for more time? The print-out in terminal is as below.(BEVFusion385) root@ubuntu:/sdd1/bevfusion# CUDA_VISIBLE_DEVICE=7 torchpack dist-run -np 4 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox [ubuntu:2322777] pmix_mca_base_component_repository_open: unable to open mca_pnet_test: perhaps a missing symbol, or compiled for a different version of OpenPMIx (ignored) [ubuntu:2322777] pmix_mca_base_component_repository_open: unable to open mca_pnet_tcp: perhaps a missing symbol, or compiled for a different version of OpenPMIx (ignored) Config (path: configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml): {'seed': 0, 'deterministic': False, 'checkpoint_config': {'interval': 1, 'max_keep_ckpts': 1}, 'log_config': {'interval': 50, 'hooks': [{'type': 'TextLoggerHook'}, {'type': 'TensorboardLoggerHook'}]}, 'load_from': None, 'resume_from': None, 'cudnn_benchmark': False, 'fp16': {'loss_scale': {'growth_interval': 2000}}, 'max_epochs': 20, 'runner': {'type': 'CustomEpochBasedRunner', 'max_epochs': 20}, 'dataset_type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'gt_paste_stop_epoch': -1, 'reduce_beams': 32, 'load_dim': 5, 'use_dim': 5, 'load_augmented': None, 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'voxel_size': [0.075, 0.075, 0.2], 'image_size': [256, 704], 'augment2d': {'resize': [[0.38, 0.55], [0.48, 0.48]], 'rotate': [-5.4, 5.4], 'gridmask': {'prob': 0.0, 'fixed_prob': True}}, 'augment3d': {'scale': [0.9, 1.1], 'rotate': [-0.78539816, 0.78539816], 'translate': 0.5}, 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'input_modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'train_pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ObjectPaste', 'stop_epoch': -1, 'db_sampler': {'dataset_root': 'dataset/NuScenes/', 'info_path': 'dataset/NuScenes/nuscenes_dbinfos_train.pkl', 'rate': 1.0, 'prepare': {'filter_by_difficulty': [-1], 'filter_by_min_points': {'car': 5, 'truck': 5, 'bus': 5, 'trailer': 5, 'construction_vehicle': 5, 'traffic_cone': 5, 'barrier': 5, 'motorcycle': 5, 'bicycle': 5, 'pedestrian': 5}}, 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'sample_groups': {'car': 2, 'truck': 3, 'construction_vehicle': 7, 'bus': 4, 'trailer': 6, 'barrier': 2, 'motorcycle': 6, 'bicycle': 6, 'pedestrian': 2, 'traffic_cone': 2}, 'points_loader': {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32}}}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.38, 0.55], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [-5.4, 5.4], 'rand_flip': True, 'is_train': True}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [0.9, 1.1], 'rot_lim': [-0.78539816, 0.78539816], 'trans_lim': 0.5, 'is_train': True}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'RandomFlip3D'}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectNameFilter', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'GridMask', 'use_h': True, 'use_w': True, 'max_epoch': 20, 'rotate': 1, 'offset': False, 'ratio': 0.5, 'mode': 1, 'prob': 0.0, 'fixed_prob': True}, {'type': 'PointShuffle'}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'test_pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'data': {'samples_per_gpu': 4, 'workers_per_gpu': 4, 'train': {'type': 'CBGSDataset', 'dataset': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_train.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ObjectPaste', 'stop_epoch': -1, 'db_sampler': {'dataset_root': 'dataset/NuScenes/', 'info_path': 'dataset/NuScenes/nuscenes_dbinfos_train.pkl', 'rate': 1.0, 'prepare': {'filter_by_difficulty': [-1], 'filter_by_min_points': {'car': 5, 'truck': 5, 'bus': 5, 'trailer': 5, 'construction_vehicle': 5, 'traffic_cone': 5, 'barrier': 5, 'motorcycle': 5, 'bicycle': 5, 'pedestrian': 5}}, 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'sample_groups': {'car': 2, 'truck': 3, 'construction_vehicle': 7, 'bus': 4, 'trailer': 6, 'barrier': 2, 'motorcycle': 6, 'bicycle': 6, 'pedestrian': 2, 'traffic_cone': 2}, 'points_loader': {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32}}}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.38, 0.55], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [-5.4, 5.4], 'rand_flip': True, 'is_train': True}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [0.9, 1.1], 'rot_lim': [-0.78539816, 0.78539816], 'trans_lim': 0.5, 'is_train': True}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'RandomFlip3D'}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectNameFilter', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'GridMask', 'use_h': True, 'use_w': True, 'max_epoch': 20, 'rotate': 1, 'offset': False, 'ratio': 0.5, 'mode': 1, 'prob': 0.0, 'fixed_prob': True}, {'type': 'PointShuffle'}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': False, 'use_valid_flag': True, 'box_type_3d': 'LiDAR'}}, 'val': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_val.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': False, 'box_type_3d': 'LiDAR'}, 'test': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_val.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': True, 'box_type_3d': 'LiDAR'}}, 'evaluation': {'interval': 1, 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}]}, 'model': {'type': 'BEVFusion', 'encoders': {'camera': {'neck': {'type': 'GeneralizedLSSFPN', 'in_channels': [192, 384, 768], 'out_channels': 256, 'start_level': 0, 'num_outs': 3, 'norm_cfg': {'type': 'BN2d', 'requires_grad': True}, 'act_cfg': {'type': 'ReLU', 'inplace': True}, 'upsample_cfg': {'mode': 'bilinear', 'align_corners': False}}, 'vtransform': {'type': 'DepthLSSTransform', 'in_channels': 256, 'out_channels': 80, 'image_size': [256, 704], 'feature_size': [32, 88], 'xbound': [-54.0, 54.0, 0.3], 'ybound': [-54.0, 54.0, 0.3], 'zbound': [-10.0, 10.0, 20.0], 'dbound': [1.0, 60.0, 0.5], 'downsample': 2}, 'backbone': {'type': 'SwinTransformer', 'embed_dims': 96, 'depths': [2, 2, 6, 2], 'num_heads': [3, 6, 12, 24], 'window_size': 7, 'mlp_ratio': 4, 'qkv_bias': True, 'qk_scale': None, 'drop_rate': 0.0, 'attn_drop_rate': 0.0, 'drop_path_rate': 0.2, 'patch_norm': True, 'out_indices': [1, 2, 3], 'with_cp': False, 'convert_weights': True, 'init_cfg': {'type': 'Pretrained', 'checkpoint': 'https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth'}}}, 'lidar': {'voxelize': {'max_num_points': 10, 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'voxel_size': [0.075, 0.075, 0.2], 'max_voxels': [120000, 160000]}, 'backbone': {'type': 'SparseEncoder', 'in_channels': 5, 'sparse_shape': [1440, 1440, 41], 'output_channels': 128, 'order': ['conv', 'norm', 'act'], 'encoder_channels': [[16, 16, 32], [32, 32, 64], [64, 64, 128], [128, 128]], 'encoder_paddings': [[0, 0, 1], [0, 0, 1], [0, 0, [1, 1, 0]], [0, 0]], 'block_type': 'basicblock'}}}, 'fuser': {'type': 'ConvFuser', 'in_channels': [80, 256], 'out_channels': 256}, 'heads': {'map': None, 'object': {'type': 'TransFusionHead', 'num_proposals': 200, 'auxiliary': True, 'in_channels': 512, 'hidden_channel': 128, 'num_classes': 10, 'num_decoder_layers': 1, 'num_heads': 8, 'nms_kernel_size': 3, 'ffn_channel': 256, 'dropout': 0.1, 'bn_momentum': 0.1, 'activation': 'relu', 'train_cfg': {'dataset': 'nuScenes', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'grid_size': [1440, 1440, 41], 'voxel_size': [0.075, 0.075, 0.2], 'out_size_factor': 8, 'gaussian_overlap': 0.1, 'min_radius': 2, 'pos_weight': -1, 'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2], 'assigner': {'type': 'HungarianAssigner3D', 'iou_calculator': {'type': 'BboxOverlaps3D', 'coordinate': 'lidar'}, 'cls_cost': {'type': 'FocalLossCost', 'gamma': 2.0, 'alpha': 0.25, 'weight': 0.15}, 'reg_cost': {'type': 'BBoxBEVL1Cost', 'weight': 0.25}, 'iou_cost': {'type': 'IoU3DCost', 'weight': 0.25}}}, 'test_cfg': {'dataset': 'nuScenes', 'grid_size': [1440, 1440, 41], 'out_size_factor': 8, 'voxel_size': [0.075, 0.075], 'pc_range': [-54.0, -54.0], 'nms_type': None}, 'common_heads': {'center': [2, 2], 'height': [1, 2], 'dim': [3, 2], 'rot': [2, 2], 'vel': [2, 2]}, 'bbox_coder': {'type': 'TransFusionBBoxCoder', 'pc_range': [-54.0, -54.0], 'post_center_range': [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0], 'score_threshold': 0.0, 'out_size_factor': 8, 'voxel_size': [0.075, 0.075], 'code_size': 10}, 'loss_cls': {'type': 'FocalLoss', 'use_sigmoid': True, 'gamma': 2.0, 'alpha': 0.25, 'reduction': 'mean', 'loss_weight': 1.0}, 'loss_heatmap': {'type': 'GaussianFocalLoss', 'reduction': 'mean', 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'reduction': 'mean', 'loss_weight': 0.25}}}, 'decoder': {'backbone': {'type': 'SECOND', 'in_channels': 256, 'out_channels': [128, 256], 'layer_nums': [5, 5], 'layer_strides': [1, 2], 'norm_cfg': {'type': 'BN', 'eps': 0.001, 'momentum': 0.01}, 'conv_cfg': {'type': 'Conv2d', 'bias': False}}, 'neck': {'type': 'SECONDFPN', 'in_channels': [128, 256], 'out_channels': [256, 256], 'upsample_strides': [1, 2], 'norm_cfg': {'type': 'BN', 'eps': 0.001, 'momentum': 0.01}, 'upsample_cfg': {'type': 'deconv', 'bias': False}, 'use_conv_for_no_stride': True}}}, 'optimizer': {'type': 'AdamW', 'lr': 0.0001, 'weight_decay': 0.01}, 'optimizer_config': {'grad_clip': {'max_norm': 35, 'norm_type': 2}}, 'lr_config': {'policy': 'cyclic'}, 'momentum_config': {'policy': 'cyclic'}} dict_keys(['type', 'dataset_root', 'ann_file', 'pipeline', 'object_classes', 'map_classes', 'modality', 'test_mode', 'box_type_3d']) /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 2022-11-28 10:55:16,597 - mmdet - INFO - load checkpoint from http path: https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth load checkpoint from local path: pretrained/bevfusion-det.pth [ ] 0/6019, elapsed: 0s, ETA:/root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /sdd1/bevfusion/mmdet3d/models/heads/bbox/transfusion.py:273: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). top_proposals_class = top_proposals // heatmap.shape[-1] [ ] 4/6019, 0.2 task/s, elapsed: 19s, ETA: 27922s/sdd1/bevfusion/mmdet3d/core/bbox/coders/transfusion_bbox_coder.py:96: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). self.post_center_range = torch.tensor( [>>] 6020/6019, 8.6 task/s, elapsed: 700s, ETA: 0sWhen I watch my GPU device, instead of the 8th one, the device activated actually are 1st, 3rd and 4th one. All of them are 100% used while the program seems to stuck. Did you make a mistake here? Your code is for a single GPU, so it should be "np 1"
LeongVan
commented
1 year ago Thanks for your kind reply. Yes, that's a mistake. Now I notice that. I'll try your instruction to install a new environment to see whether everything can work as expected.
LeongVan
commented
1 year ago I also encountered the same problem. I modified part of the code in LSS, and the debugging can be successful, but it will get stuck at this place. If I use the original code that has not been modified, everything is normal
I didn't changed the original code, but still met the problem. Yesterday, I trained on 4 GPUs, and when I checked out this morning, it stuck again. I'm wondering if it's a problem with the environment installation. This is the process of my environment installation. If I don't modify any code, there will be no problem running. You can refer to it
Thanks a lot!
I set up a brand new environment under your instruction, and it works except a little error when running python setup.py develop as AttributeError: module 'distutils' has no attribute 'version'.
To solve this, I follow the solution here, which is to pip install setuptools==59.5.0, and then run setup.py and everything works!
LeongVan
commented
1 year ago Hi, thanks for opening source program. I've got a question. I was running the evaluation using command
CUDA_VISIBLE_DEVICE=7 torchpack dist-run -np 4 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox(CUDA_VISIBLE_DEVICE=7 is used to control which GPU device is activated, '7' means I merely used the 8th one). But I got the following situation: the program seems to stuck without any process running. I've been waiting for half an hour and no process was going on. Is it normal or should I waiting for more time? The print-out in terminal is as below.(BEVFusion385) root@ubuntu:/sdd1/bevfusion# CUDA_VISIBLE_DEVICE=7 torchpack dist-run -np 4 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox [ubuntu:2322777] pmix_mca_base_component_repository_open: unable to open mca_pnet_test: perhaps a missing symbol, or compiled for a different version of OpenPMIx (ignored) [ubuntu:2322777] pmix_mca_base_component_repository_open: unable to open mca_pnet_tcp: perhaps a missing symbol, or compiled for a different version of OpenPMIx (ignored) Config (path: configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml): {'seed': 0, 'deterministic': False, 'checkpoint_config': {'interval': 1, 'max_keep_ckpts': 1}, 'log_config': {'interval': 50, 'hooks': [{'type': 'TextLoggerHook'}, {'type': 'TensorboardLoggerHook'}]}, 'load_from': None, 'resume_from': None, 'cudnn_benchmark': False, 'fp16': {'loss_scale': {'growth_interval': 2000}}, 'max_epochs': 20, 'runner': {'type': 'CustomEpochBasedRunner', 'max_epochs': 20}, 'dataset_type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'gt_paste_stop_epoch': -1, 'reduce_beams': 32, 'load_dim': 5, 'use_dim': 5, 'load_augmented': None, 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'voxel_size': [0.075, 0.075, 0.2], 'image_size': [256, 704], 'augment2d': {'resize': [[0.38, 0.55], [0.48, 0.48]], 'rotate': [-5.4, 5.4], 'gridmask': {'prob': 0.0, 'fixed_prob': True}}, 'augment3d': {'scale': [0.9, 1.1], 'rotate': [-0.78539816, 0.78539816], 'translate': 0.5}, 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'input_modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'train_pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ObjectPaste', 'stop_epoch': -1, 'db_sampler': {'dataset_root': 'dataset/NuScenes/', 'info_path': 'dataset/NuScenes/nuscenes_dbinfos_train.pkl', 'rate': 1.0, 'prepare': {'filter_by_difficulty': [-1], 'filter_by_min_points': {'car': 5, 'truck': 5, 'bus': 5, 'trailer': 5, 'construction_vehicle': 5, 'traffic_cone': 5, 'barrier': 5, 'motorcycle': 5, 'bicycle': 5, 'pedestrian': 5}}, 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'sample_groups': {'car': 2, 'truck': 3, 'construction_vehicle': 7, 'bus': 4, 'trailer': 6, 'barrier': 2, 'motorcycle': 6, 'bicycle': 6, 'pedestrian': 2, 'traffic_cone': 2}, 'points_loader': {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32}}}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.38, 0.55], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [-5.4, 5.4], 'rand_flip': True, 'is_train': True}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [0.9, 1.1], 'rot_lim': [-0.78539816, 0.78539816], 'trans_lim': 0.5, 'is_train': True}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'RandomFlip3D'}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectNameFilter', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'GridMask', 'use_h': True, 'use_w': True, 'max_epoch': 20, 'rotate': 1, 'offset': False, 'ratio': 0.5, 'mode': 1, 'prob': 0.0, 'fixed_prob': True}, {'type': 'PointShuffle'}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'test_pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'data': {'samples_per_gpu': 4, 'workers_per_gpu': 4, 'train': {'type': 'CBGSDataset', 'dataset': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_train.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ObjectPaste', 'stop_epoch': -1, 'db_sampler': {'dataset_root': 'dataset/NuScenes/', 'info_path': 'dataset/NuScenes/nuscenes_dbinfos_train.pkl', 'rate': 1.0, 'prepare': {'filter_by_difficulty': [-1], 'filter_by_min_points': {'car': 5, 'truck': 5, 'bus': 5, 'trailer': 5, 'construction_vehicle': 5, 'traffic_cone': 5, 'barrier': 5, 'motorcycle': 5, 'bicycle': 5, 'pedestrian': 5}}, 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'sample_groups': {'car': 2, 'truck': 3, 'construction_vehicle': 7, 'bus': 4, 'trailer': 6, 'barrier': 2, 'motorcycle': 6, 'bicycle': 6, 'pedestrian': 2, 'traffic_cone': 2}, 'points_loader': {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32}}}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.38, 0.55], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [-5.4, 5.4], 'rand_flip': True, 'is_train': True}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [0.9, 1.1], 'rot_lim': [-0.78539816, 0.78539816], 'trans_lim': 0.5, 'is_train': True}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'RandomFlip3D'}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ObjectNameFilter', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'GridMask', 'use_h': True, 'use_w': True, 'max_epoch': 20, 'rotate': 1, 'offset': False, 'ratio': 0.5, 'mode': 1, 'prob': 0.0, 'fixed_prob': True}, {'type': 'PointShuffle'}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': False, 'use_valid_flag': True, 'box_type_3d': 'LiDAR'}}, 'val': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_val.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': False, 'box_type_3d': 'LiDAR'}, 'test': {'type': 'NuScenesDataset', 'dataset_root': 'dataset/NuScenes/', 'ann_file': 'dataset/NuScenes/nuscenes_infos_val.pkl', 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}], 'object_classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone'], 'map_classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider'], 'modality': {'use_lidar': True, 'use_camera': True, 'use_radar': False, 'use_map': False, 'use_external': False}, 'test_mode': True, 'box_type_3d': 'LiDAR'}}, 'evaluation': {'interval': 1, 'pipeline': [{'type': 'LoadMultiViewImageFromFiles', 'to_float32': True}, {'type': 'LoadPointsFromFile', 'coord_type': 'LIDAR', 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'load_augmented': None}, {'type': 'LoadPointsFromMultiSweeps', 'sweeps_num': 9, 'load_dim': 5, 'use_dim': 5, 'reduce_beams': 32, 'pad_empty_sweeps': True, 'remove_close': True, 'load_augmented': None}, {'type': 'LoadAnnotations3D', 'with_bbox_3d': True, 'with_label_3d': True, 'with_attr_label': False}, {'type': 'ImageAug3D', 'final_dim': [256, 704], 'resize_lim': [0.48, 0.48], 'bot_pct_lim': [0.0, 0.0], 'rot_lim': [0.0, 0.0], 'rand_flip': False, 'is_train': False}, {'type': 'GlobalRotScaleTrans', 'resize_lim': [1.0, 1.0], 'rot_lim': [0.0, 0.0], 'trans_lim': 0.0, 'is_train': False}, {'type': 'LoadBEVSegmentation', 'dataset_root': 'dataset/NuScenes/', 'xbound': [-50.0, 50.0, 0.5], 'ybound': [-50.0, 50.0, 0.5], 'classes': ['drivable_area', 'ped_crossing', 'walkway', 'stop_line', 'carpark_area', 'divider']}, {'type': 'PointsRangeFilter', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0]}, {'type': 'ImageNormalize', 'mean': [0.485, 0.456, 0.406], 'std': [0.229, 0.224, 0.225]}, {'type': 'DefaultFormatBundle3D', 'classes': ['car', 'truck', 'construction_vehicle', 'bus', 'trailer', 'barrier', 'motorcycle', 'bicycle', 'pedestrian', 'traffic_cone']}, {'type': 'Collect3D', 'keys': ['img', 'points', 'gt_bboxes_3d', 'gt_labels_3d', 'gt_masks_bev'], 'meta_keys': ['camera_intrinsics', 'camera2ego', 'lidar2ego', 'lidar2camera', 'camera2lidar', 'lidar2image', 'img_aug_matrix', 'lidar_aug_matrix']}]}, 'model': {'type': 'BEVFusion', 'encoders': {'camera': {'neck': {'type': 'GeneralizedLSSFPN', 'in_channels': [192, 384, 768], 'out_channels': 256, 'start_level': 0, 'num_outs': 3, 'norm_cfg': {'type': 'BN2d', 'requires_grad': True}, 'act_cfg': {'type': 'ReLU', 'inplace': True}, 'upsample_cfg': {'mode': 'bilinear', 'align_corners': False}}, 'vtransform': {'type': 'DepthLSSTransform', 'in_channels': 256, 'out_channels': 80, 'image_size': [256, 704], 'feature_size': [32, 88], 'xbound': [-54.0, 54.0, 0.3], 'ybound': [-54.0, 54.0, 0.3], 'zbound': [-10.0, 10.0, 20.0], 'dbound': [1.0, 60.0, 0.5], 'downsample': 2}, 'backbone': {'type': 'SwinTransformer', 'embed_dims': 96, 'depths': [2, 2, 6, 2], 'num_heads': [3, 6, 12, 24], 'window_size': 7, 'mlp_ratio': 4, 'qkv_bias': True, 'qk_scale': None, 'drop_rate': 0.0, 'attn_drop_rate': 0.0, 'drop_path_rate': 0.2, 'patch_norm': True, 'out_indices': [1, 2, 3], 'with_cp': False, 'convert_weights': True, 'init_cfg': {'type': 'Pretrained', 'checkpoint': 'https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth'}}}, 'lidar': {'voxelize': {'max_num_points': 10, 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'voxel_size': [0.075, 0.075, 0.2], 'max_voxels': [120000, 160000]}, 'backbone': {'type': 'SparseEncoder', 'in_channels': 5, 'sparse_shape': [1440, 1440, 41], 'output_channels': 128, 'order': ['conv', 'norm', 'act'], 'encoder_channels': [[16, 16, 32], [32, 32, 64], [64, 64, 128], [128, 128]], 'encoder_paddings': [[0, 0, 1], [0, 0, 1], [0, 0, [1, 1, 0]], [0, 0]], 'block_type': 'basicblock'}}}, 'fuser': {'type': 'ConvFuser', 'in_channels': [80, 256], 'out_channels': 256}, 'heads': {'map': None, 'object': {'type': 'TransFusionHead', 'num_proposals': 200, 'auxiliary': True, 'in_channels': 512, 'hidden_channel': 128, 'num_classes': 10, 'num_decoder_layers': 1, 'num_heads': 8, 'nms_kernel_size': 3, 'ffn_channel': 256, 'dropout': 0.1, 'bn_momentum': 0.1, 'activation': 'relu', 'train_cfg': {'dataset': 'nuScenes', 'point_cloud_range': [-54.0, -54.0, -5.0, 54.0, 54.0, 3.0], 'grid_size': [1440, 1440, 41], 'voxel_size': [0.075, 0.075, 0.2], 'out_size_factor': 8, 'gaussian_overlap': 0.1, 'min_radius': 2, 'pos_weight': -1, 'code_weights': [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.2, 0.2], 'assigner': {'type': 'HungarianAssigner3D', 'iou_calculator': {'type': 'BboxOverlaps3D', 'coordinate': 'lidar'}, 'cls_cost': {'type': 'FocalLossCost', 'gamma': 2.0, 'alpha': 0.25, 'weight': 0.15}, 'reg_cost': {'type': 'BBoxBEVL1Cost', 'weight': 0.25}, 'iou_cost': {'type': 'IoU3DCost', 'weight': 0.25}}}, 'test_cfg': {'dataset': 'nuScenes', 'grid_size': [1440, 1440, 41], 'out_size_factor': 8, 'voxel_size': [0.075, 0.075], 'pc_range': [-54.0, -54.0], 'nms_type': None}, 'common_heads': {'center': [2, 2], 'height': [1, 2], 'dim': [3, 2], 'rot': [2, 2], 'vel': [2, 2]}, 'bbox_coder': {'type': 'TransFusionBBoxCoder', 'pc_range': [-54.0, -54.0], 'post_center_range': [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0], 'score_threshold': 0.0, 'out_size_factor': 8, 'voxel_size': [0.075, 0.075], 'code_size': 10}, 'loss_cls': {'type': 'FocalLoss', 'use_sigmoid': True, 'gamma': 2.0, 'alpha': 0.25, 'reduction': 'mean', 'loss_weight': 1.0}, 'loss_heatmap': {'type': 'GaussianFocalLoss', 'reduction': 'mean', 'loss_weight': 1.0}, 'loss_bbox': {'type': 'L1Loss', 'reduction': 'mean', 'loss_weight': 0.25}}}, 'decoder': {'backbone': {'type': 'SECOND', 'in_channels': 256, 'out_channels': [128, 256], 'layer_nums': [5, 5], 'layer_strides': [1, 2], 'norm_cfg': {'type': 'BN', 'eps': 0.001, 'momentum': 0.01}, 'conv_cfg': {'type': 'Conv2d', 'bias': False}}, 'neck': {'type': 'SECONDFPN', 'in_channels': [128, 256], 'out_channels': [256, 256], 'upsample_strides': [1, 2], 'norm_cfg': {'type': 'BN', 'eps': 0.001, 'momentum': 0.01}, 'upsample_cfg': {'type': 'deconv', 'bias': False}, 'use_conv_for_no_stride': True}}}, 'optimizer': {'type': 'AdamW', 'lr': 0.0001, 'weight_decay': 0.01}, 'optimizer_config': {'grad_clip': {'max_norm': 35, 'norm_type': 2}}, 'lr_config': {'policy': 'cyclic'}, 'momentum_config': {'policy': 'cyclic'}} dict_keys(['type', 'dataset_root', 'ann_file', 'pipeline', 'object_classes', 'map_classes', 'modality', 'test_mode', 'box_type_3d']) /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1639180588308/work/aten/src/ATen/native/TensorShape.cpp:2157.) return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined] 2022-11-28 10:55:16,597 - mmdet - INFO - load checkpoint from http path: https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_tiny_patch4_window7_224.pth load checkpoint from local path: pretrained/bevfusion-det.pth [ ] 0/6019, elapsed: 0s, ETA:/root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. exteriors = [int_coords(poly.exterior.coords) for poly in polygons] /root/anaconda3/envs/BEVFusion385/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry. interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors] /sdd1/bevfusion/mmdet3d/models/heads/bbox/transfusion.py:273: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor'). top_proposals_class = top_proposals // heatmap.shape[-1] [ ] 4/6019, 0.2 task/s, elapsed: 19s, ETA: 27922s/sdd1/bevfusion/mmdet3d/core/bbox/coders/transfusion_bbox_coder.py:96: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor). self.post_center_range = torch.tensor( [>>] 6020/6019, 8.6 task/s, elapsed: 700s, ETA: 0sWhen I watch my GPU device, instead of the 8th one, the device activated actually are 1st, 3rd and 4th one. All of them are 100% used while the program seems to stuck. Did you make a mistake here? Your code is for a single GPU, so it should be "np 1"
我按照你的指导，结合我环境的实际情况，按照如下步骤搭建了我的虚拟环境。
####################
# Environment: cuda=11.7, Ubuntu 20.04 #
####################
# Create and Activate Virtual Environment
conda create -n bevf38 python=3.8
conda activate bevf38
# Install Packages
conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge
pip install opencv-python
pip install mmcv-full==1.4.0
pip install mmdet==2.20.0
pip install Pillow==8.4.0
pip install tqdm
pip install torchpack
pip install nuscenes-devkit
pip install setuptools==59.5.0
conda install openmpi
# Set Environment Variables
'''
vim ~/.bashrc
add the following line in the last line of .bashrc file
OMPI_MCA_opal_cuda_support=true
'''
# Install Package
conda install mpi4py
pip install ninja
pip install numba
# source environment
source ~/.bashrc
# conpile the setup.py (firstly activate virtual environment after source)
python setup.py develop但是当我跑训练的时候，有些命令能顺利跑起来，比如用相机做检测，
CUDA_VISIBLE_DEVICES=2,3,4 torchpack dist-run -np 3 -v python tools/train.py configs/nuscenes/det/centerhead/lssfpn/camera/256x704/swint/default.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth，
或者用相机做分割，
CUDA_VISIBLE_DEVICES=1,2,3,4 torchpack dist-run -np 4 -v python tools/train.py configs/nuscenes/seg/camera-bev256d2.yaml，
但是有些命令跑不起来，比如用雷达做检测，
CUDA_VISIBLE_DEVICES=1,2,3,4 torchpack dist-run -np 4 -v python tools/train.py configs/nuscenes/det/transfusion/secfpn/lidar/voxelnet_0p075.yaml，
或者用雷达+相机做分割，
CUDA_VISIBLE_DEVICES=2,3,4 torchpack dist-run -np 3 -v python tools/train.py configs/nuscenes/seg/fusion-bev256d2-lss-custom.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth。
他们报的错都是一样的，如下所示。
2022-12-08 16:57:21,871 - mmdet3d - INFO - workflow: [('train', 1)], max: 20 epochs
2022-12-08 16:57:21,873 - mmdet3d - INFO - Checkpoints will be saved to /sdd1/bevfusion/runs/run-da751a2c-f2c16e29 by HardDiskBackend.
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1634272068694/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1634272068694/work/aten/src/ATen/native/TensorShape.cpp:2157.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/sdd1/bevfusion/mmdet3d/models/heads/bbox/transfusion.py:273: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
top_proposals_class = top_proposals // heatmap.shape[-1]
/sdd1/bevfusion/mmdet3d/models/heads/bbox/transfusion.py:535: UserWarning: __floordiv__ is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
grid_size[:2] // self.train_cfg["out_size_factor"]
2022-12-08 16:57:31,518 - mmcv - INFO - Reducer buckets have been rebuilt in this iteration.
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
exteriors = [int_coords(poly.exterior.coords) for poly in polygons]
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors]
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1823: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
exteriors = [int_coords(poly.exterior.coords) for poly in polygons]
/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/nuscenes/map_expansion/map_api.py:1824: ShapelyDeprecationWarning: Iteration over multi-part geometries is deprecated and will be removed in Shapely 2.0. Use the `geoms` property to access the constituent parts of a multi-part geometry.
interiors = [int_coords(pi.coords) for poly in polygons for pi in poly.interiors]
Traceback (most recent call last):
File "tools/train.py", line 91, in <module>
main()
File "tools/train.py", line 80, in main
train_model(
File "/sdd1/bevfusion/mmdet3d/apis/train.py", line 126, in train_model
runner.run(data_loaders, [("train", 1)])
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 127, in run
epoch_runner(data_loaders[i], **kwargs)
File "/sdd1/bevfusion/mmdet3d/runner/epoch_based_runner.py", line 14, in train
super().train(data_loader, **kwargs)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 50, in train
self.run_iter(data_batch, train_mode=True, **kwargs)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/epoch_based_runner.py", line 29, in run_iter
outputs = self.model.train_step(data_batch, self.optimizer,
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/parallel/distributed.py", line 52, in train_step
output = self.module.train_step(*inputs[0], **kwargs[0])
File "/sdd1/bevfusion/mmdet3d/models/fusion_models/base.py", line 78, in train_step
losses = self(**data)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/sdd1/bevfusion/mmdet3d/models/fusion_models/bevfusion.py", line 187, in forward
outputs = self.forward_single(
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/sdd1/bevfusion/mmdet3d/models/fusion_models/bevfusion.py", line 246, in forward_single
feature = self.extract_lidar_features(points)
File "/sdd1/bevfusion/mmdet3d/models/fusion_models/bevfusion.py", line 132, in extract_lidar_features
x = self.encoders["lidar"]["backbone"](feats, coords, batch_size, sizes=sizes)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/mmcv/runner/fp16_utils.py", line 128, in new_func
output = old_func(*new_args, **new_kwargs)
File "/sdd1/bevfusion/mmdet3d/models/backbones/sparse_encoder.py", line 120, in forward
x = encoder_layer(x)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/sdd1/bevfusion/mmdet3d/ops/spconv/modules.py", line 132, in forward
input = module(input)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/sdd1/bevfusion/mmdet3d/ops/spconv/modules.py", line 132, in forward
input = module(input)
File "/root/anaconda3/envs/bevf38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
return forward_call(*input, **kwargs)
File "/sdd1/bevfusion/mmdet3d/ops/spconv/conv.py", line 163, in forward
outids, indice_pairs, indice_pair_num = ops.get_indice_pairs(
File "/sdd1/bevfusion/mmdet3d/ops/spconv/ops.py", line 92, in get_indice_pairs
return get_indice_pairs_func(
RuntimeError: /sdd1/bevfusion/mmdet3d/ops/spconv/include/tensorview/helper_launch.h 17
N > 0 assert faild. CUDA kernel launch blocks must be positive, but got N= 0请问你有遇到过类似的问题吗？知道其中的原因吗？如果有，请问你是怎么解决的呢？
lilkeker
commented
1 year ago I have not encountered a similar problem. I run the camera plus lidar detection test now. There is no problem. I think this is caused by a wrong environment installation. You can try "pip install mmcv-full==1.4.0 - f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html", because I see that you are not the same as me when installing mmcv, if not, I don’t know how to solve it up
LeongVan
commented
1 year ago Thanks for your advice. I have tried both pip install mmcv-full==1.4.0 and pip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html. Unfortunately, none of them works. Same bug appears.
lilkeker
commented
1 year ago Thanks for your advice. I have tried both
pip install mmcv-full==1.4.0andpip install mmcv-full==1.4.0 -f https://download.openmmlab.com/mmcv/dist/cu113/torch1.10.0/index.html. Unfortunately, none of them works. Same bug appears.
Sorry, I can't solve the problem you are currently encountering, if you solve it, please let me know your solution
LeongVan
commented
1 year ago Now the problem is more clear. If I run on single one or two GPUs, the program runs successfully. Instead, if I run on more than 2 GPUs, such as 3, 4, or more, the problem mentioned above is triggered. Such a strange bug.
kentang-mit
commented
1 year ago Sorry for missing the discussion. I think you might want to check whether you used some flags like CUDA_VISIBLE_DEVICES in your command. If you specified that only two GPUs are visible and you want to use -np 4 it might not work.
LeongVan
commented
1 year ago Sorry for missing the discussion. I think you might want to check whether you used some flags like
CUDA_VISIBLE_DEVICESin your command. If you specified that only two GPUs are visible and you want to use-np 4it might not work.
Thanks for your reply.
Actually I've noticed that little typo already, so the running cammands afterwards I used is something like CUDA_VISIBLE_DEVICES=1,2,3 torchpack dist-run -np 3 -v python tools/train.py configs/nuscenes/det/transfusion/secfpn/lidar/voxelnet_0p075.yam, where -np is consistent with the number of GPUs I used. But the same bug appear as well, as printed in the terminal, N > 0 assert faild. CUDA kernel launch blocks must be positive, but got N= 0.
Also, I checked the discussion in #28, but found no solution to this bug. It seems like a problem related to spconv block.
Any clue will be appreciate.
kentang-mit
commented
1 year ago Did you have a chance to check out the answer in https://github.com/mit-han-lab/bevfusion/issues/28#issuecomment-1160333858? It seems that you are using CUDA 11.7, but you are trying to configure the CUDA environment in conda to be 11.3. I can confirm that the current version of this codebase requires the PyTorch version to be 1.10.x, and probably there is no CUDA11.7-compiled PyTorch wheel available. So my suggestion will be to install cuda-11.3 on your local machine, temporarily symbolic link /usr/local/cuda to /usr/local/cuda-11.3 and try again.
By the way, we are currently working on a new version of this codebase without the spconv backend. It will take some time, please stay tuned.
LeongVan
commented
1 year ago Thanks for your suggestion.
I tried CUDA 11.3 as well, but it didn't work... The CUDA in system is 11.7 (base environment), and I installed CUDA 11.3 in my virtual environment.
This time, the running command I used was CUDA_VISIBLE_DEVICES=4,5,6,7 torchpack dist-run -np 4 -v python tools/train.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml --model.encoders.camera.backbone.init_cfg.checkpoint pretrained/swint-nuimages-pretrained.pth --load_from pretrained/lidar-only-det.pth, which is suggested in README.md.
kentang-mit
commented
1 year ago Yes, I understand your problem, and I just want to confirm whether you tried to install CUDA 11.3 on the ubuntu system (not in the anaconda virtual environment). According to the discussions in issue 28, it seems that your CUDA version in the OS should match with the CUDA version in anaconda.
I knew that the environment setup could be painful, and we are working hard to make it easier.
lilkeker
commented
1 year ago I still have this problem, when I ran the program today, it got stuck again,
###############################################################
(packnetsan) lk@user-SYS-7049GP-TRT:~/bevfusion_ori$ nvidia-smi
Fri Dec 23 13:06:16 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 465.19.01 Driver Version: 465.19.01 CUDA Version: 11.3 |
|-------------------------------+----------------------+----------------------+
################################################################# packnetsan) lk@user-SYS-7049GP-TRT:~/bevfusion_ori$ python Python 3.8.13 (default, Oct 21 2022, 23:50:54) [GCC 11.2.0] :: Anaconda, Inc. on linux Type "help", "copyright", "credits" or "license" for more information.
import torch torch.version '1.10.0+cu113' torch.version.cuda '11.3' torch.backends.cudnn.version() 8200 exit()
kentang-mit
commented
1 year ago Hi @lilkeker,
Did you have a chance to add a -v flag to the command you are running and see if there will be any error messages?
Best, Haotian
lilkeker
commented
1 year ago Hi @lilkeker,
Did you have a chance to add a
-vflag to the command you are running and see if there will be any error messages?Best, Haotian Thanks a lot ,Haotian, I think I have found the reason, it is because I modified a part of the code,and the code will be stuck
kentang-mit
commented
1 year ago Hi @lilkeker,
It seems to me that there are some custom modules that are not implemented by us in your codebase. What the error means is that the input and weight to your convolution layer (self.conv_base in the error log) are not on the same GPU. You may need to check what happened in this function call in your custom modules.
Best, Haotian
youngfly
commented
1 year ago I met the problem, too. But when I test on a single gpu, this issue seems disappear. Pretty strange.
kentang-mit
commented
1 year ago Closed due to inactivity. Please feel free to reopen if you feel it necessary.
ZoeSnow
commented
10 months ago I find a solution and it works, maybe you can try it. Reference https://zhuanlan.zhihu.com/p/319888724, it is caused by the types of CPU and GPU
Hi, thanks for opening source program. I've got a question. I was running the evaluation using command
CUDA_VISIBLE_DEVICE=7 torchpack dist-run -np 4 python tools/test.py configs/nuscenes/det/transfusion/secfpn/camera+lidar/swint_v0p075/convfuser.yaml pretrained/bevfusion-det.pth --eval bbox(CUDA_VISIBLE_DEVICE=7 is used to control which GPU device is activated, '7' means I merely used the 8th one). But I got the following situation: the program seems to stuck without any process running. I've been waiting for half an hour and no process was going on. Is it normal or should I waiting for more time? The print-out in terminal is as below.When I watch my GPU device, instead of the 8th one, the device activated actually are 1st, 3rd and 4th one. All of them are 100% used while the program seems to stuck.