suvasis commented 2 years ago

hi,

I have 2 gpu machine. Single GPU training works ok. However for 2 gpus, It seems to be waiting for something for ever. The GPU utilization is 0% for both GPUs.

I am using Kitti dataset as described in the documentation.

The command as shown:

OpenPCDet/tools$ ./scripts/dist_train.sh 2 --batch_size 2 --epochs 1 --cfg_file cfgs/kitti_models/pv_rcnn.yaml

My code base is as of Sep 22, 2022.

Log snippet:

2022-09-23 10:21:21,551 INFO **Start training kitti_models/pv_rcnn(default)** epochs: 0it [00:07, ?it/s] 2022-09-23 10:21:30,211 INFO **End training kitti_models/pv_rcnn(default)**

2022-09-23 10:21:30,212 INFO **Start evaluation kitti_models/pv_rcnn(default)** 2022-09-23 10:21:30,213 INFO Loading KITTI dataset 2022-09-23 10:21:30,304 INFO Total samples for KITTI dataset: 3769 Wait 30 seconds for next check (progress: 0.0 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/default/ckpt

LOG SNIPPET:

NGPUS=2

PY_ARGS='--batch_size 2 --epochs 1 --cfg_file cfgs/kitti_models/pv_rcnn.yaml'
true
PORT=27915 ++ nc -z 127.0.0.1 27915 ++ echo 1
status=1
'[' 1 '!=' 0 ']'
break
echo 27915 27915
echo --batch_size 2 --epochs 1 --cfg_file cfgs/kitti_models/pv_rcnn.yaml --batch_size 2 --epochs 1 --cfg_file cfgs/kitti_models/pv_rcnn.yaml
python -m torch.distributed.launch --nproc_per_node=2 --rdzv_endpoint=localhost:27915 train.py --launcher pytorch --batch_size 2 --epochs 1 --cfg_file cfgs/kitti_models/pv_rcnn.yaml /home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launch.py:178: FutureWarning: The module torch.distributed.launch is deprecated and will be removed in future. Use torchrun. Note that --use_env is set by default in torchrun. If your script expects --local_rank argument to be set, please change it to read from os.environ['LOCAL_RANK'] instead. See https://pytorch.org/docs/stable/distributed.html#launch-utility for further instructions

warnings.warn( WARNING:torch.distributed.run:

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

2022-09-23 10:21:20,169 INFO **Start logging** 2022-09-23 10:21:20,169 INFO CUDA_VISIBLE_DEVICES=ALL 2022-09-23 10:21:20,169 INFO total_batch_size: 2 2022-09-23 10:21:20,169 INFO cfg_file cfgs/kitti_models/pv_rcnn.yaml 2022-09-23 10:21:20,169 INFO batch_size 1 2022-09-23 10:21:20,170 INFO epochs 1 2022-09-23 10:21:20,170 INFO workers 4 2022-09-23 10:21:20,170 INFO extra_tag default 2022-09-23 10:21:20,170 INFO ckpt None 2022-09-23 10:21:20,170 INFO pretrained_model None 2022-09-23 10:21:20,170 INFO launcher pytorch 2022-09-23 10:21:20,170 INFO tcp_port 18888 2022-09-23 10:21:20,170 INFO sync_bn False 2022-09-23 10:21:20,170 INFO fix_random_seed False 2022-09-23 10:21:20,170 INFO ckpt_save_interval 1 2022-09-23 10:21:20,170 INFO local_rank 0 2022-09-23 10:21:20,170 INFO max_ckpt_save_num 30 2022-09-23 10:21:20,170 INFO merge_all_iters_to_one_epoch False 2022-09-23 10:21:20,170 INFO set_cfgs None 2022-09-23 10:21:20,170 INFO max_waiting_mins 0 2022-09-23 10:21:20,170 INFO start_epoch 0 2022-09-23 10:21:20,170 INFO num_epochs_to_eval 0 2022-09-23 10:21:20,170 INFO save_to_file False 2022-09-23 10:21:20,170 INFO use_tqdm_to_record False 2022-09-23 10:21:20,170 INFO logger_iter_interval 50 2022-09-23 10:21:20,170 INFO ckpt_save_time_interval 300 2022-09-23 10:21:20,170 INFO wo_gpu_stat False 2022-09-23 10:21:20,170 INFO cfg.ROOT_DIR: /home/minasm/suvasis/tools/pvrcnn/OpenPCDet 2022-09-23 10:21:20,170 INFO cfg.LOCAL_RANK: 0 2022-09-23 10:21:20,170 INFO cfg.CLASS_NAMES: ['Car', 'Pedestrian', 'Cyclist'] 2022-09-23 10:21:20,170 INFO cfg.DATA_CONFIG = edict() 2022-09-23 10:21:20,170 INFO cfg.DATA_CONFIG.DATASET: KittiDataset 2022-09-23 10:21:20,170 INFO cfg.DATA_CONFIG.DATA_PATH: ../data/kitti 2022-09-23 10:21:20,170 INFO cfg.DATA_CONFIG.POINT_CLOUD_RANGE: [0, -40, -3, 70.4, 40, 1] 2022-09-23 10:21:20,170 INFO cfg.DATA_CONFIG.DATA_SPLIT = edict() 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_SPLIT.train: train 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_SPLIT.test: val 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.INFO_PATH = edict() 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.INFO_PATH.train: ['kitti_infos_train.pkl'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.INFO_PATH.test: ['kitti_infos_val.pkl'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.GET_ITEM_LIST: ['points'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.FOV_POINTS_ONLY: True 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR = edict() 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.DISABLE_AUG_LIST: ['placeholder'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_AUGMENTOR.AUG_CONFIG_LIST: [{'NAME': 'gt_sampling', 'USE_ROAD_PLANE': True, 'DB_INFO_PATH': ['kitti_dbinfos_train.pkl'], 'PREPARE': {'filter_by_min_points': ['Car:5', 'Pedestrian:5', 'Cyclist:5'], 'filter_by_difficulty': [-1]}, 'SAMPLE_GROUPS': ['Car:15', 'Pedestrian:10', 'Cyclist:10'], 'NUM_POINT_FEATURES': 4, 'DATABASE_WITH_FAKELIDAR': False, 'REMOVE_EXTRA_WIDTH': [0.0, 0.0, 0.0], 'LIMIT_WHOLE_SCENE': False}, {'NAME': 'random_world_flip', 'ALONG_AXIS_LIST': ['x']}, {'NAME': 'random_world_rotation', 'WORLD_ROT_ANGLE': [-0.78539816, 0.78539816]}, {'NAME': 'random_world_scaling', 'WORLD_SCALE_RANGE': [0.95, 1.05]}] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING = edict() 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.encoding_type: absolute_coordinates_encoding 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.used_feature_list: ['x', 'y', 'z', 'intensity'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.POINT_FEATURE_ENCODING.src_feature_list: ['x', 'y', 'z', 'intensity'] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG.DATA_PROCESSOR: [{'NAME': 'mask_points_and_boxes_outside_range', 'REMOVE_OUTSIDE_BOXES': True}, {'NAME': 'shuffle_points', 'SHUFFLE_ENABLED': {'train': True, 'test': False}}, {'NAME': 'transform_points_to_voxels', 'VOXEL_SIZE': [0.05, 0.05, 0.1], 'MAX_POINTS_PER_VOXEL': 5, 'MAX_NUMBER_OF_VOXELS': {'train': 16000, 'test': 40000}}] 2022-09-23 10:21:20,171 INFO cfg.DATA_CONFIG._BASECONFIG: cfgs/dataset_configs/kitti_dataset.yaml 2022-09-23 10:21:20,171 INFO cfg.MODEL = edict() 2022-09-23 10:21:20,171 INFO cfg.MODEL.NAME: PVRCNN 2022-09-23 10:21:20,171 INFO cfg.MODEL.VFE = edict() 2022-09-23 10:21:20,171 INFO cfg.MODEL.VFE.NAME: MeanVFE 2022-09-23 10:21:20,171 INFO cfg.MODEL.BACKBONE_3D = edict() 2022-09-23 10:21:20,171 INFO cfg.MODEL.BACKBONE_3D.NAME: VoxelBackBone8x 2022-09-23 10:21:20,171 INFO cfg.MODEL.MAP_TO_BEV = edict() 2022-09-23 10:21:20,171 INFO cfg.MODEL.MAP_TO_BEV.NAME: HeightCompression 2022-09-23 10:21:20,171 INFO cfg.MODEL.MAP_TO_BEV.NUM_BEV_FEATURES: 256 2022-09-23 10:21:20,171 INFO cfg.MODEL.BACKBONE_2D = edict() 2022-09-23 10:21:20,171 INFO cfg.MODEL.BACKBONE_2D.NAME: BaseBEVBackbone 2022-09-23 10:21:20,171 INFO cfg.MODEL.BACKBONE_2D.LAYER_NUMS: [5, 5] 2022-09-23 10:21:20,172 INFO cfg.MODEL.BACKBONE_2D.LAYER_STRIDES: [1, 2] 2022-09-23 10:21:20,172 INFO cfg.MODEL.BACKBONE_2D.NUM_FILTERS: [128, 256] 2022-09-23 10:21:20,172 INFO cfg.MODEL.BACKBONE_2D.UPSAMPLE_STRIDES: [1, 2] 2022-09-23 10:21:20,172 INFO cfg.MODEL.BACKBONE_2D.NUM_UPSAMPLE_FILTERS: [256, 256] 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD = edict() 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.NAME: AnchorHeadSingle 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.CLASS_AGNOSTIC: False 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.USE_DIRECTION_CLASSIFIER: True 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.DIR_OFFSET: 0.78539 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.DIR_LIMIT_OFFSET: 0.0 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.NUM_DIR_BINS: 2 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.ANCHOR_GENERATOR_CONFIG: [{'class_name': 'Car', 'anchor_sizes': [[3.9, 1.6, 1.56]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-1.78], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.6, 'unmatched_threshold': 0.45}, {'class_name': 'Pedestrian', 'anchor_sizes': [[0.8, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}, {'class_name': 'Cyclist', 'anchor_sizes': [[1.76, 0.6, 1.73]], 'anchor_rotations': [0, 1.57], 'anchor_bottom_heights': [-0.6], 'align_center': False, 'feature_map_stride': 8, 'matched_threshold': 0.5, 'unmatched_threshold': 0.35}] 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG = edict() 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NAME: AxisAlignedTargetAssigner 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.POS_FRACTION: -1.0 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.SAMPLE_SIZE: 512 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.NORM_BY_NUM_EXAMPLES: False 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.MATCH_HEIGHT: False 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.TARGET_ASSIGNER_CONFIG.BOX_CODER: ResidualCoder 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG = edict() 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.cls_weight: 1.0 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.loc_weight: 2.0 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.dir_weight: 0.2 2022-09-23 10:21:20,172 INFO cfg.MODEL.DENSE_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 2022-09-23 10:21:20,172 INFO cfg.MODEL.PFE = edict() 2022-09-23 10:21:20,172 INFO cfg.MODEL.PFE.NAME: VoxelSetAbstraction 2022-09-23 10:21:20,172 INFO cfg.MODEL.PFE.POINT_SOURCE: raw_points 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.NUM_KEYPOINTS: 2048 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.NUM_OUTPUT_FEATURES: 128 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SAMPLE_METHOD: FPS 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.FEATURES_SOURCE: ['bev', 'x_conv1', 'x_conv2', 'x_conv3', 'x_conv4', 'raw_points'] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.raw_points = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.raw_points.MLPS: [[16, 16], [16, 16]] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.raw_points.POOL_RADIUS: [0.4, 0.8] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.raw_points.NSAMPLE: [16, 16] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv1 = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv1.DOWNSAMPLE_FACTOR: 1 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv1.MLPS: [[16, 16], [16, 16]] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv1.POOL_RADIUS: [0.4, 0.8] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv1.NSAMPLE: [16, 16] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv2 = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv2.DOWNSAMPLE_FACTOR: 2 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv2.MLPS: [[32, 32], [32, 32]] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv2.POOL_RADIUS: [0.8, 1.2] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv2.NSAMPLE: [16, 32] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv3 = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv3.DOWNSAMPLE_FACTOR: 4 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv3.MLPS: [[64, 64], [64, 64]] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv3.POOL_RADIUS: [1.2, 2.4] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv3.NSAMPLE: [16, 32] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv4 = edict() 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv4.DOWNSAMPLE_FACTOR: 8 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv4.MLPS: [[64, 64], [64, 64]] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv4.POOL_RADIUS: [2.4, 4.8] 2022-09-23 10:21:20,173 INFO cfg.MODEL.PFE.SA_LAYER.x_conv4.NSAMPLE: [16, 32] 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.NAME: PointHeadSimple 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.CLS_FC: [256, 256] 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.CLASS_AGNOSTIC: True 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.USE_POINT_FEATURES_BEFORE_FUSION: True 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.TARGET_CONFIG = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.TARGET_CONFIG.GT_EXTRA_WIDTH: [0.2, 0.2, 0.2] 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_REG: smooth-l1 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.POINT_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.point_cls_weight: 1.0 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NAME: PVRCNNHead 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.CLASS_AGNOSTIC: True 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.SHARED_FC: [256, 256] 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.CLS_FC: [256, 256] 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.REG_FC: [256, 256] 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.DP_RATIO: 0.3 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_TYPE: nms_gpu 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.MULTI_CLASSES_NMS: False 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_PRE_MAXSIZE: 9000 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_POST_MAXSIZE: 512 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TRAIN.NMS_THRESH: 0.8 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST = edict() 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_TYPE: nms_gpu 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.MULTI_CLASSES_NMS: False 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_PRE_MAXSIZE: 1024 2022-09-23 10:21:20,174 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_POST_MAXSIZE: 100 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.NMS_CONFIG.TEST.NMS_THRESH: 0.7 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL = edict() 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.GRID_SIZE: 6 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.MLPS: [[64, 64], [64, 64]] 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_RADIUS: [0.8, 1.6] 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.NSAMPLE: [16, 16] 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.ROI_GRID_POOL.POOL_METHOD: max_pool 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG = edict() 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.BOX_CODER: ResidualCoder 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.ROI_PER_IMAGE: 128 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.FG_RATIO: 0.5 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.SAMPLE_ROI_BY_EACH_CLASS: True 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_SCORE_TYPE: roi_iou 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_FG_THRESH: 0.75 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH: 0.25 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.CLS_BG_THRESH_LO: 0.1 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.HARD_BG_RATIO: 0.8 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.TARGET_CONFIG.REG_FG_THRESH: 0.55 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG = edict() 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CLS_LOSS: BinaryCrossEntropy 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.REG_LOSS: smooth-l1 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.CORNER_LOSS_REGULARIZATION: True 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS = edict() 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_cls_weight: 1.0 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_reg_weight: 1.0 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.rcnn_corner_weight: 1.0 2022-09-23 10:21:20,175 INFO cfg.MODEL.ROI_HEAD.LOSS_CONFIG.LOSS_WEIGHTS.code_weights: [1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0] 2022-09-23 10:21:20,175 INFO cfg.MODEL.POST_PROCESSING = edict() 2022-09-23 10:21:20,175 INFO cfg.MODEL.POST_PROCESSING.RECALL_THRESH_LIST: [0.3, 0.5, 0.7] 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.SCORE_THRESH: 0.1 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.OUTPUT_RAW_SCORE: False 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.EVAL_METRIC: kitti 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG = edict() 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.MULTI_CLASSES_NMS: False 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_TYPE: nms_gpu 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_THRESH: 0.1 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_PRE_MAXSIZE: 4096 2022-09-23 10:21:20,176 INFO cfg.MODEL.POST_PROCESSING.NMS_CONFIG.NMS_POST_MAXSIZE: 500 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION = edict() 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.BATCH_SIZE_PER_GPU: 2 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.NUM_EPOCHS: 80 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.OPTIMIZER: adam_onecycle 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.LR: 0.01 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.WEIGHT_DECAY: 0.01 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.MOMENTUM: 0.9 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.MOMS: [0.95, 0.85] 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.PCT_START: 0.4 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.DIV_FACTOR: 10 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.DECAY_STEP_LIST: [35, 45] 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.LR_DECAY: 0.1 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.LR_CLIP: 1e-07 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.LR_WARMUP: False 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.WARMUP_EPOCH: 1 2022-09-23 10:21:20,176 INFO cfg.OPTIMIZATION.GRAD_NORM_CLIP: 10 2022-09-23 10:21:20,176 INFO cfg.TAG: pv_rcnn 2022-09-23 10:21:20,176 INFO cfg.EXP_GROUP_PATH: kitti_models 2022-09-23 10:21:20,273 INFO Database filter by min points Car: 14357 => 13532 2022-09-23 10:21:20,273 INFO Database filter by min points Pedestrian: 2207 => 2168 2022-09-23 10:21:20,273 INFO Database filter by min points Cyclist: 734 => 705 2022-09-23 10:21:20,289 INFO Database filter by difficulty Car: 13532 => 10759 2022-09-23 10:21:20,291 INFO Database filter by difficulty Pedestrian: 2168 => 2075 2022-09-23 10:21:20,292 INFO Database filter by difficulty Cyclist: 705 => 581 2022-09-23 10:21:20,296 INFO Loading KITTI dataset 2022-09-23 10:21:20,370 INFO Total samples for KITTI dataset: 3712 /home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1659484806139/work/aten/src/ATen/native/TensorShape.cpp:2894.) return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] /home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/functional.py:478: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /opt/conda/conda-bld/pytorch_1659484806139/work/aten/src/ATen/native/TensorShape.cpp:2894.) return _VF.meshgrid(tensors, kwargs) # type: ignore[attr-defined] 2022-09-23 10:21:21,303 INFO ==> Loading parameters from checkpoint /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/default/ckpt/checkpoint_epoch_2.pth to CPU 2022-09-23 10:21:21,413 INFO ==> Loading optimizer parameters from checkpoint /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/default/ckpt/checkpoint_epoch_2.pth to CPU ==> Checkpoint trained from version: pcdet+0.6.0+b61049f 2022-09-23 10:21:21,495 INFO ==> Done ==> Checkpoint trained from version: pcdet+0.6.0+b61049f 2022-09-23 10:21:21,548 INFO DistributedDataParallel( (module): PVRCNN( (vfe): MeanVFE() (backbone_3d): VoxelBackBone8x( (conv_input): SparseSequential( (0): SubMConv3d(4, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (conv1): SparseSequential( (0): SparseSequential( (0): SubMConv3d(16, 16, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(16, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv2): SparseSequential( (0): SparseSequential( (0): SparseConv3d(16, 32, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d(32, 32, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(32, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv3): SparseSequential( (0): SparseSequential( (0): SparseConv3d(32, 64, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[1, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv4): SparseSequential( (0): SparseSequential( (0): SparseConv3d(64, 64, kernel_size=[3, 3, 3], stride=[2, 2, 2], padding=[0, 1, 1], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): SparseSequential( (0): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (2): SparseSequential( (0): SubMConv3d(64, 64, kernel_size=[3, 3, 3], stride=[1, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(64, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (conv_out): SparseSequential( (0): SparseConv3d(64, 128, kernel_size=[3, 1, 1], stride=[2, 1, 1], padding=[0, 0, 0], dilation=[1, 1, 1], output_padding=[0, 0, 0], bias=False, algo=ConvAlgo.MaskImplicitGemm) (1): BatchNorm1d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) (map_to_bev_module): HeightCompression() (pfe): VoxelSetAbstraction( (SA_layers): ModuleList( (0): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(19, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(19, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) (1): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(35, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(35, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(32, 32, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(32, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) (2): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) (3): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(67, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) ) (SA_rawpoints): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(4, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(4, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(16, 16, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(16, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) (vsa_point_feature_fusion): Sequential( (0): Linear(in_features=640, out_features=128, bias=False) (1): BatchNorm1d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() ) ) (backbone_2d): BaseBEVBackbone( (blocks): ModuleList( (0): Sequential( (0): ZeroPad2d((1, 1, 1, 1)) (1): Conv2d(256, 128, kernel_size=(3, 3), stride=(1, 1), bias=False) (2): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(128, 128, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(128, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) (1): Sequential( (0): ZeroPad2d((1, 1, 1, 1)) (1): Conv2d(128, 256, kernel_size=(3, 3), stride=(2, 2), bias=False) (2): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (3): ReLU() (4): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (5): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (6): ReLU() (7): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (8): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (9): ReLU() (10): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (11): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (12): ReLU() (13): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (14): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (15): ReLU() (16): Conv2d(256, 256, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False) (17): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (18): ReLU() ) ) (deblocks): ModuleList( (0): Sequential( (0): ConvTranspose2d(128, 256, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) (1): Sequential( (0): ConvTranspose2d(256, 256, kernel_size=(2, 2), stride=(2, 2), bias=False) (1): BatchNorm2d(256, eps=0.001, momentum=0.01, affine=True, track_running_stats=True) (2): ReLU() ) ) ) (dense_head): AnchorHeadSingle( (cls_loss_func): SigmoidFocalClassificationLoss() (reg_loss_func): WeightedSmoothL1Loss() (dir_loss_func): WeightedCrossEntropyLoss() (conv_cls): Conv2d(512, 18, kernel_size=(1, 1), stride=(1, 1)) (conv_box): Conv2d(512, 42, kernel_size=(1, 1), stride=(1, 1)) (conv_dir_cls): Conv2d(512, 12, kernel_size=(1, 1), stride=(1, 1)) ) (point_head): PointHeadSimple( (cls_loss_func): SigmoidFocalClassificationLoss() (cls_layers): Sequential( (0): Linear(in_features=640, out_features=256, bias=False) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Linear(in_features=256, out_features=256, bias=False) (4): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() (6): Linear(in_features=256, out_features=1, bias=True) ) ) (roi_head): PVRCNNHead( (proposal_target_layer): ProposalTargetLayer() (reg_loss_func): WeightedSmoothL1Loss() (roi_grid_pool_layer): StackSAModuleMSG( (groupers): ModuleList( (0): QueryAndGroup() (1): QueryAndGroup() ) (mlps): ModuleList( (0): Sequential( (0): Conv2d(131, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) (1): Sequential( (0): Conv2d(131, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Conv2d(64, 64, kernel_size=(1, 1), stride=(1, 1), bias=False) (4): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (5): ReLU() ) ) ) (shared_fc_layer): Sequential( (0): Conv1d(27648, 256, kernel_size=(1,), stride=(1,), bias=False) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Dropout(p=0.3, inplace=False) (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False) (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (6): ReLU() ) (cls_layers): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Dropout(p=0.3, inplace=False) (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False) (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (6): ReLU() (7): Conv1d(256, 1, kernel_size=(1,), stride=(1,)) ) (reg_layers): Sequential( (0): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False) (1): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (2): ReLU() (3): Dropout(p=0.3, inplace=False) (4): Conv1d(256, 256, kernel_size=(1,), stride=(1,), bias=False) (5): BatchNorm1d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True) (6): ReLU() (7): Conv1d(256, 7, kernel_size=(1,), stride=(1,)) ) ) ) ) 2022-09-23 10:21:21,551 INFO **Start training kitti_models/pv_rcnn(default)** epochs: 0it [00:07, ?it/s] 2022-09-23 10:21:30,211 INFO **End training kitti_models/pv_rcnn(default)**

2022-09-23 10:21:30,212 INFO **Start evaluation kitti_models/pv_rcnn(default)** 2022-09-23 10:21:30,213 INFO Loading KITTI dataset 2022-09-23 10:21:30,304 INFO Total samples for KITTI dataset: 3769 Wait 30 seconds for next check (progress: 0.0 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/defaulWait 30 seconds for next check (progress: 0.5 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/defaulWait 30 seconds for next check (progress: 1.0 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/defaulWait 30 seconds for next check (progress: 1.5 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/defaul

jihanyang commented 2 years ago

2022-09-23 10:21:21,551 INFO Start training kitti_models/pv_rcnn(default)
epochs: 0it [00:07, ?it/s]
2022-09-23 10:21:30,211 INFO End training kitti_models/pv_rcnn(default)

As shown in your log, the training has been finished in previous lauch.

suvasis commented 2 years ago

hi,

After I followed this comment https://github.com/open-mmlab/OpenPCDet/issues/938

Have you tried to comment these two lines to quickly start the training?

#if mp.get_start_method(allow_none=True) is None:
#    mp.set_start_method('spawn')

The training worked only for batch_size 2. If I increase the batch_size to 4 or beyond (the log snippet is attached below). What should I do to fix this?

For batch_size=2, the run the successful. For batch_size=4, the training fails.

///////////////////////////////////////////////////////////////////////////////////////////

batch_size=2

///////////////////////////////////////////////////////////////////////////////////////////

command: (pytorchbuild) minasm@lambda-quad:~/suvasis/tools/pvrcnn/OpenPCDet/tools$ ./scripts/dist_train.sh 2 --batch_size 2 --epochs 10 --cfg_file cfgs/kitti_models/pv_rcnn.yaml

log: Cyclist AP@0.50, 0.25, 0.25: bbox AP:90.7043, 78.4946, 74.1905 bev AP:90.0927, 74.9485, 70.5419 3d AP:90.0911, 74.9443, 70.5419 aos AP:90.47, 76.53, 72.11 Cyclist AP_R40@0.50, 0.25, 0.25: bbox AP:94.0771, 79.2091, 75.3161 bev AP:93.6473, 75.5676, 72.0880 3d AP:93.6469, 75.5664, 72.0693 aos AP:93.83, 77.14, 73.01

2022-09-23 14:07:09,004 INFO Result is save to /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/default/eval/eval_with_train/epoch_10/val 2022-09-23 14:07:09,005 INFO ****Evaluation done.***** 2022-09-23 14:07:09,026 INFO Epoch 10 has been evaluated Wait 30 seconds for next check (progress: 0.0 / 0 minutes): /home/minasm/suvasis/tools/pvrcnn/OpenPCDet/output/kitti_models/pv_rcnn/defaul2022-09-23 14:07:39,058 INFO **End evaluation kitti_models/pv_rcnn(default)** ///////////////////////////////////////////////////////////////////////////////////////////

batch_size=4

///////////////////////////////////////////////////////////////////////////////////////////

command: OpenPCDet/tools$ ./scripts/dist_train.sh 2 --batch_size 4 --epochs 10 --cfg_file cfgs/kitti_models/pv_rcnn.yaml

Log:

pochs: 0%| | 0/10 [00:00<?, ?it/s]2022-09-23 17:46:02,631 INFO Start training kitti_models/pv_rcnn(default) epochs: 0%| | 0/10 [00:00<?, ?it/s/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/autograd/init.py:173: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance. grad.sizes() = [7, 256, 1], strides() = [256, 1, 256] bucket_view.sizes() = [7, 256, 1], strides() = [256, 1, 1] (Triggered internally at /opt/conda/conda-bld/pytorch_1659484806139/work/torch/csrc/distributed/c10d/reducer.cpp:312.) Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass /home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/autograd/init.py:173: UserWarning: Grad strides do not match bucket view strides. This may indicate grad was not created according to the gradient layout contract, or that the param's strides changed since DDP was constructed. This is not an error, but may impair performance. grad.sizes() = [7, 256, 1], strides() = [256, 1, 256] bucket_view.sizes() = [7, 256, 1], strides() = [256, 1, 1] (Triggered internally at /opt/conda/conda-bld/pytorch_1659484806139/work/torch/csrc/distributed/c10d/reducer.cpp:312.) Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2022-09-23 17:46:09,141 INFO epoch: 0/10, acc_iter=1, cur_iter=0/928, batch_size=2, time_cost(epoch): 00:06/1:35:24, time_cost(all): 00:06/15:54:01, loss=9.989889144897461, d_time=1.04(1.04), f_time=5.11(5.11), b_time=6.15(6.15), lr=0.0009999999999999992 Traceback (most recent call last): File "/home/minasm/suvasis/tools/pvrcnn/OpenPCDet/tools/train.py", line 221, in main() File "/home/minasm/suvasis/tools/pvrcnn/OpenPCDet/tools/train.py", line 168, in main train_model( File "/home/minasm/suvasis/tools/pvrcnn/OpenPCDet/tools/train_utils/train_utils.py", line 150, in train_model accumulated_iter = train_one_epoch( File "/home/minasm/suvasis/tools/pvrcnn/OpenPCDet/tools/train_utils/train_utils.py", line 54, in train_one_epoch loss.backward() File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/_tensor.py", line 396, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs) File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/autograd/init.py", line 173, in backward Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass RuntimeError: CUDA out of memory. Tried to allocate 444.00 MiB (GPU 1; 7.80 GiB total capacity; 4.73 GiB already allocated; 418.81 MiB free; 5.81 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 97166 closing signal SIGTERM ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 1 (pid: 97167) of binary: /home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/bin/python Traceback (most recent call last): File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/runpy.py", line 197, in _run_module_as_main return _run_code(code, main_globals, None, File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/runpy.py", line 87, in _run_code exec(code, run_globals) File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launch.py", line 193, in main() File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launch.py", line 189, in main launch(args) File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launch.py", line 174, in launch run(args) File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/run.py", line 752, in run elastic_launch( File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 131, in call return launch_agent(self._config, self._entrypoint, list(args)) File "/home/minasm/suvasis/tools/anaconda3/envs/pytorchbuild/lib/python3.9/site-packages/torch/distributed/launcher/api.py", line 245, in launch_agent raise ChildFailedError( torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

train.py FAILED

Failures:

------------------------------------------------------------ Root Cause (first observed failure): [0]: time : 2022-09-23_17:46:15 host : lambda-quad rank : 1 (local_rank: 1) exitcode : 1 (pid: 97167) error_file: traceback : To enable traceback see: https://pytorch.org/docs/stable/elastic/errors.html ============================================================ ///////////////////////////machine details/////////////////// nvidia-smi Fri Sep 23 17:50:22 2022 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 515.65.01 Driver Version: 515.65.01 CUDA Version: 11.7 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A | | 30% 39C P8 15W / 220W | 67MiB / 8192MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ | 1 NVIDIA GeForce ... On | 00000000:21:00.0 Off | N/A | | 30% 38C P8 20W / 220W | 5MiB / 8192MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 1733 G /usr/lib/xorg/Xorg 56MiB | | 0 N/A N/A 1908 G /usr/bin/gnome-shell 8MiB | | 1 N/A N/A 1733 G /usr/lib/xorg/Xorg 4MiB | +-----------------------------------------------------------------------------+ tools$ nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2022 NVIDIA Corporation Built on Wed_Jun__8_16:49:14_PDT_2022 Cuda compilation tools, release 11.7, V11.7.99 Build cuda_11.7.r11.7/compiler.31442593_0

github-actions[bot] commented 1 year ago

This issue is stale because it has been open for 30 days with no activity.

open-mmlab / OpenPCDet

training on multiple gpu #1122

Have you tried to comment these two lines to quickly start the training?

train.py FAILED