EPNet++: Cascade Bi-directional Fusion for Multi-Modal 3D Object Detection (TPAMI-2022)
MIT License
56
stars
6
forks
source link
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)` #15
Hi, when I try to reproduce the code I get an error, my CUDA version is 10.0 and torch version is 1.2.0. can someone please tell me how I should go about fixing this error. Thanks.
/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/config.py:250: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
yaml_cfg = edict(yaml.load(f))
1.0 1.0
2023-11-04 13:34:47,355 INFO **Start logging**
2023-11-04 13:34:47,355 INFO CUDA_VISIBLE_DEVICES=ALL
2023-11-04 13:34:47,355 INFO cfg_file cfgs/CAR_EPNet_plus_plus.yaml
2023-11-04 13:34:47,355 INFO train_mode rcnn_online
2023-11-04 13:34:47,356 INFO batch_size 16
2023-11-04 13:34:47,356 INFO epochs 50
2023-11-04 13:34:47,356 INFO workers 8
2023-11-04 13:34:47,356 INFO ckpt_save_interval 1
2023-11-04 13:34:47,356 INFO output_dir ./log/CAR_EPNet_plus_plus/
2023-11-04 13:34:47,356 INFO mgpus True
2023-11-04 13:34:47,356 INFO data_path ../../data/
2023-11-04 13:34:47,356 INFO ckpt None
2023-11-04 13:34:47,356 INFO rpn_ckpt None
2023-11-04 13:34:47,356 INFO gt_database None
2023-11-04 13:34:47,356 INFO rcnn_training_roi_dir None
2023-11-04 13:34:47,356 INFO rcnn_training_feature_dir None
2023-11-04 13:34:47,356 INFO train_with_eval False
2023-11-04 13:34:47,356 INFO rcnn_eval_roi_dir None
2023-11-04 13:34:47,356 INFO rcnn_eval_feature_dir None
2023-11-04 13:34:47,356 INFO set_cfgs ['LI_FUSION.ENABLED', 'True', 'LI_FUSION.ADD_Image_Attention', 'True', 'CROSS_FUSION', 'True', 'USE_P2I_GATE', 'True', 'DEEP_RCNN_FUSION', 'False', 'USE_IMAGE_LOSS', 'True', 'IMAGE_WEIGHT', '1.0', 'USE_IMAGE_SCORE', 'True', 'USE_IMG_DENSE_LOSS', 'True', 'USE_MC_LOSS', 'True', 'MC_LOSS_WEIGHT', '1.0', 'I2P_Weight', '0.5', 'P2I_Weight', '0.5', 'ADD_MC_MASK', 'True', 'MC_MASK_THRES', '0.2', 'SAVE_MODEL_PREP', '0.8']
2023-11-04 13:34:47,356 INFO model_type base
2023-11-04 13:34:47,356 INFO cfg.TAG: CAR_EPNet_plus_plus
2023-11-04 13:34:47,357 INFO cfg.CLASSES: Car
2023-11-04 13:34:47,357 INFO cfg.INCLUDE_SIMILAR_TYPE: True
2023-11-04 13:34:47,357 INFO cfg.AUG_DATA: True
2023-11-04 13:34:47,357 INFO cfg.AUG_METHOD_LIST: ['rotation', 'scaling', 'flip']
2023-11-04 13:34:47,357 INFO cfg.AUG_METHOD_PROB: [1.0, 1.0, 0.5]
2023-11-04 13:34:47,357 INFO cfg.AUG_ROT_RANGE: 18
2023-11-04 13:34:47,357 INFO cfg.GT_AUG_ENABLED: False
2023-11-04 13:34:47,357 INFO cfg.GT_EXTRA_NUM: 15
2023-11-04 13:34:47,357 INFO cfg.GT_AUG_RAND_NUM: True
2023-11-04 13:34:47,357 INFO cfg.GT_AUG_APPLY_PROB: 1.0
2023-11-04 13:34:47,357 INFO cfg.GT_AUG_HARD_RATIO: 0.6
2023-11-04 13:34:47,357 INFO cfg.PC_REDUCE_BY_RANGE: True
2023-11-04 13:34:47,357 INFO cfg.PC_AREA_SCOPE: [[-40. 40. ]
[ -1. 3. ]
[ 0. 70.4]]
2023-11-04 13:34:47,358 INFO cfg.CLS_MEAN_SIZE: [[1.5256319 1.6285675 3.8831165]]
2023-11-04 13:34:47,358 INFO cfg.USE_IOU_BRANCH: True
2023-11-04 13:34:47,358 INFO cfg.USE_IM_DEPTH: False
2023-11-04 13:34:47,358 INFO cfg.USE_PSEUDO_LIDAR: False
2023-11-04 13:34:47,358 INFO cfg.CROSS_FUSION: True
2023-11-04 13:34:47,358 INFO cfg.INPUT_CROSS_FUSION: False
2023-11-04 13:34:47,358 INFO cfg.USE_KNN_FUSION: False
2023-11-04 13:34:47,358 INFO cfg.USE_SELF_ATTENTION: False
2023-11-04 13:34:47,358 INFO cfg.DEEP_RCNN_FUSION: False
2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_LOSS: True
2023-11-04 13:34:47,358 INFO cfg.IMAGE_WEIGHT: 1.0
2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_LOSS_TYPE: CrossEntropyLoss
2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_SCORE: True
2023-11-04 13:34:47,358 INFO cfg.USE_IMG_DENSE_LOSS: True
2023-11-04 13:34:47,358 INFO cfg.USE_KL_LOSS: False
2023-11-04 13:34:47,358 INFO cfg.USE_KL_LOSS_TYPE: KL
2023-11-04 13:34:47,358 INFO cfg.MC_LOSS_WEIGHT: 1.0
2023-11-04 13:34:47,359 INFO cfg.SAVE_MODEL_PREP: 0.8
2023-11-04 13:34:47,359 INFO cfg.USE_P2I_GATE: True
2023-11-04 13:34:47,359 INFO cfg.STACK_CROSS_FUSION: False
2023-11-04 13:34:47,359 INFO cfg.USE_IMAGE_RES: False
2023-11-04 13:34:47,359 INFO cfg.RCNN_IMG_CHANNEL: 32
2023-11-04 13:34:47,359 INFO cfg.ONLY_USE_IMAGE_FEAT: False
2023-11-04 13:34:47,359 INFO cfg.USE_POINT_ATT_FEATURE: False
2023-11-04 13:34:47,359 INFO cfg.USE_POINT_FEATURE_RES: False
2023-11-04 13:34:47,359 INFO cfg.I2P_Weight: 0.5
2023-11-04 13:34:47,359 INFO cfg.P2I_Weight: 0.5
2023-11-04 13:34:47,359 INFO cfg.USE_MC_LOSS: True
2023-11-04 13:34:47,359 INFO cfg.ADD_MC_MASK: True
2023-11-04 13:34:47,359 INFO cfg.MC_MASK_THRES: 0.2
2023-11-04 13:34:47,359 INFO cfg.USE_PURE_IMG_BACKBONE: False
2023-11-04 13:34:47,359 INFO cfg.USE_PAINTING_SCORE: False
2023-11-04 13:34:47,359 INFO cfg.USE_PAINTING_FEAT: False
2023-11-04 13:34:47,359 INFO
cfg.LI_FUSION = edict()
2023-11-04 13:34:47,359 INFO cfg.LI_FUSION.ENABLED: True
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.IMG_FEATURES_CHANNEL: 128
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.ADD_Image_Attention: True
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.IMG_CHANNELS: [3, 64, 128, 256, 512]
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.POINT_CHANNELS: [96, 256, 512, 1024]
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Reduce: [16, 16, 16, 16]
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Kernels: [2, 4, 8, 16]
2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Strides: [2, 4, 8, 16]
2023-11-04 13:34:47,360 INFO
cfg.RPN = edict()
2023-11-04 13:34:47,360 INFO cfg.RPN.ENABLED: True
2023-11-04 13:34:47,360 INFO cfg.RPN.FIXED: False
2023-11-04 13:34:47,360 INFO cfg.RPN.USE_INTENSITY: False
2023-11-04 13:34:47,360 INFO cfg.RPN.USE_RGB: False
2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_XZ_FINE: True
2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_SCOPE: 3.0
2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_BIN_SIZE: 0.5
2023-11-04 13:34:47,360 INFO cfg.RPN.NUM_HEAD_BIN: 12
2023-11-04 13:34:47,360 INFO cfg.RPN.BACKBONE: pointnet2_msg
2023-11-04 13:34:47,361 INFO cfg.RPN.USE_BN: True
2023-11-04 13:34:47,361 INFO cfg.RPN.NUM_POINTS: 16384
2023-11-04 13:34:47,361 INFO
cfg.RPN.SA_CONFIG = edict()
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.ATTN_DIM: 128
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.ATTN: [0, 0, 128, 128]
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.NPOINTS: [4096, 1024, 256, 64]
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.RADIUS: [[0.1, 0.5], [0.5, 1.0], [1.0, 2.0], [2.0, 4.0]]
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.NSAMPLE: [[16, 32], [16, 32], [16, 32], [16, 32]]
2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.MLPS: [[[16, 16, 32], [32, 32, 64]], [[64, 64, 128], [64, 96, 128]], [[128, 196, 256], [128, 196, 256]], [[256, 256, 512], [256, 384, 512]]]
2023-11-04 13:34:47,361 INFO cfg.RPN.FP_MLPS: [[128, 128], [256, 256], [512, 512], [512, 512]]
2023-11-04 13:34:47,361 INFO cfg.RPN.CLS_FC: [128]
2023-11-04 13:34:47,361 INFO cfg.RPN.REG_FC: [128]
2023-11-04 13:34:47,361 INFO cfg.RPN.DP_RATIO: 0.5
2023-11-04 13:34:47,361 INFO cfg.RPN.LOSS_CLS: SigmoidFocalLoss
2023-11-04 13:34:47,361 INFO cfg.RPN.FG_WEIGHT: 15
2023-11-04 13:34:47,361 INFO cfg.RPN.FOCAL_ALPHA: [0.25, 0.75]
2023-11-04 13:34:47,361 INFO cfg.RPN.FOCAL_GAMMA: 2.0
2023-11-04 13:34:47,361 INFO cfg.RPN.REG_LOSS_WEIGHT: [1.0, 1.0, 1.0, 1.0]
2023-11-04 13:34:47,362 INFO cfg.RPN.LOSS_WEIGHT: [1.0, 1.0]
2023-11-04 13:34:47,362 INFO cfg.RPN.NMS_TYPE: normal
2023-11-04 13:34:47,362 INFO cfg.RPN.SCORE_THRESH: 0.2
2023-11-04 13:34:47,362 INFO
cfg.RCNN = edict()
2023-11-04 13:34:47,362 INFO cfg.RCNN.ENABLED: True
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_RPN_FEATURES: True
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_MASK: True
2023-11-04 13:34:47,362 INFO cfg.RCNN.MASK_TYPE: seg
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_INTENSITY: False
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_DEPTH: True
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_SEG_SCORE: False
2023-11-04 13:34:47,362 INFO cfg.RCNN.ROI_SAMPLE_JIT: True
2023-11-04 13:34:47,362 INFO cfg.RCNN.ROI_FG_AUG_TIMES: 10
2023-11-04 13:34:47,362 INFO cfg.RCNN.REG_AUG_METHOD: multiple
2023-11-04 13:34:47,362 INFO cfg.RCNN.POOL_EXTRA_WIDTH: 0.2
2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_RGB: False
2023-11-04 13:34:47,362 INFO cfg.RCNN.LOC_SCOPE: 1.5
2023-11-04 13:34:47,362 INFO cfg.RCNN.LOC_BIN_SIZE: 0.5
2023-11-04 13:34:47,363 INFO cfg.RCNN.NUM_HEAD_BIN: 9
2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_BY_BIN: False
2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_SCOPE: 0.5
2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_BIN_SIZE: 0.25
2023-11-04 13:34:47,363 INFO cfg.RCNN.SIZE_RES_ON_ROI: False
2023-11-04 13:34:47,363 INFO cfg.RCNN.USE_BN: False
2023-11-04 13:34:47,363 INFO cfg.RCNN.DP_RATIO: 0.0
2023-11-04 13:34:47,363 INFO cfg.RCNN.BACKBONE: pointnet
2023-11-04 13:34:47,363 INFO cfg.RCNN.XYZ_UP_LAYER: [128, 128]
2023-11-04 13:34:47,363 INFO cfg.RCNN.NUM_POINTS: 512
2023-11-04 13:34:47,363 INFO
cfg.RCNN.SA_CONFIG = edict()
2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.NPOINTS: [128, 32, -1]
2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.RADIUS: [0.2, 0.4, 100]
2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.NSAMPLE: [64, 64, 64]
2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.MLPS: [[128, 128, 128], [128, 128, 256], [256, 256, 512]]
2023-11-04 13:34:47,363 INFO cfg.RCNN.CLS_FC: [512, 512]
2023-11-04 13:34:47,363 INFO cfg.RCNN.REG_FC: [512, 512]
2023-11-04 13:34:47,363 INFO cfg.RCNN.LOSS_CLS: BinaryCrossEntropy
2023-11-04 13:34:47,364 INFO cfg.RCNN.FOCAL_ALPHA: [0.25, 0.75]
2023-11-04 13:34:47,364 INFO cfg.RCNN.FOCAL_GAMMA: 2.0
2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_WEIGHT: [1. 1. 1.]
2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_FG_THRESH: 0.6
2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_BG_THRESH: 0.45
2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_BG_THRESH_LO: 0.05
2023-11-04 13:34:47,364 INFO cfg.RCNN.REG_FG_THRESH: 0.55
2023-11-04 13:34:47,364 INFO cfg.RCNN.FG_RATIO: 0.5
2023-11-04 13:34:47,364 INFO cfg.RCNN.ROI_PER_IMAGE: 64
2023-11-04 13:34:47,364 INFO cfg.RCNN.HARD_BG_RATIO: 0.8
2023-11-04 13:34:47,364 INFO cfg.RCNN.IOU_LOSS_TYPE: raw
2023-11-04 13:34:47,364 INFO cfg.RCNN.IOU_ANGLE_POWER: 1
2023-11-04 13:34:47,364 INFO cfg.RCNN.SCORE_THRESH: 0.2
2023-11-04 13:34:47,364 INFO cfg.RCNN.NMS_THRESH: 0.1
2023-11-04 13:34:47,364 INFO
cfg.TRAIN = edict()
2023-11-04 13:34:47,364 INFO cfg.TRAIN.SPLIT: train
2023-11-04 13:34:47,365 INFO cfg.TRAIN.VAL_SPLIT: smallval
2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR: 0.002
2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_CLIP: 1e-05
2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_DECAY: 0.5
2023-11-04 13:34:47,365 INFO cfg.TRAIN.DECAY_STEP_LIST: [100, 150, 180, 200]
2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_WARMUP: True
2023-11-04 13:34:47,365 INFO cfg.TRAIN.WARMUP_MIN: 0.0002
2023-11-04 13:34:47,365 INFO cfg.TRAIN.WARMUP_EPOCH: 1
2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_MOMENTUM: 0.1
2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_DECAY: 0.5
2023-11-04 13:34:47,365 INFO cfg.TRAIN.BNM_CLIP: 0.01
2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_DECAY_STEP_LIST: [1000]
2023-11-04 13:34:47,365 INFO cfg.TRAIN.OPTIMIZER: adam_onecycle
2023-11-04 13:34:47,365 INFO cfg.TRAIN.WEIGHT_DECAY: 0.001
2023-11-04 13:34:47,365 INFO cfg.TRAIN.MOMENTUM: 0.9
2023-11-04 13:34:47,365 INFO cfg.TRAIN.MOMS: [0.95, 0.85]
2023-11-04 13:34:47,365 INFO cfg.TRAIN.DIV_FACTOR: 10.0
2023-11-04 13:34:47,365 INFO cfg.TRAIN.PCT_START: 0.4
2023-11-04 13:34:47,366 INFO cfg.TRAIN.GRAD_NORM_CLIP: 1.0
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_PRE_NMS_TOP_N: 9000
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_POST_NMS_TOP_N: 512
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_NMS_THRESH: 0.85
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_DISTANCE_BASED_PROPOSE: True
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_TRAIN_WEIGHT: 1.0
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RCNN_TRAIN_WEIGHT: 1.0
2023-11-04 13:34:47,366 INFO cfg.TRAIN.CE_WEIGHT: 5.0
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_CE_WEIGHT: 5.0
2023-11-04 13:34:47,366 INFO cfg.TRAIN.IOU_LOSS_TYPE: cls_mask_with_bin
2023-11-04 13:34:47,366 INFO cfg.TRAIN.BBOX_AVG_BY_BIN: True
2023-11-04 13:34:47,366 INFO cfg.TRAIN.RY_WITH_BIN: False
2023-11-04 13:34:47,366 INFO
cfg.TEST = edict()
2023-11-04 13:34:47,366 INFO cfg.TEST.SPLIT: val
2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_PRE_NMS_TOP_N: 9000
2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_POST_NMS_TOP_N: 100
2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_NMS_THRESH: 0.8
2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_DISTANCE_BASED_PROPOSE: True
2023-11-04 13:34:47,366 INFO cfg.TEST.BBOX_AVG_BY_BIN: True
2023-11-04 13:34:47,366 INFO cfg.TEST.RY_WITH_BIN: False
cp: -r not specified; omitting directory '../lib/'
cp: -r not specified; omitting directory '../tools'
cp: 无法获取'../*.py' 的文件状态(stat): 没有那个文件或目录
./log/CAR_EPNet_plus_plus/
2023-11-04 13:34:47,378 INFO Loading TRAIN samples from ../../data/KITTI/object/training/label_2 ...
2023-11-04 13:34:47,892 INFO Done: filter TRAIN results: 3265 / 3712
##############USE Fusion_Cross_Conv_Gate(ADD)#########
##############ADDITION PI2 ATTENTION#########
##############USE Fusion_Cross_Conv_Gate(ADD)#########
##############ADDITION PI2 ATTENTION#########
##############USE Fusion_Cross_Conv_Gate(ADD)#########
##############ADDITION PI2 ATTENTION#########
##############USE Fusion_Cross_Conv_Gate(ADD)#########
##############ADDITION PI2 ATTENTION#########
2023-11-04 13:42:55,602 INFO **Start training**
epochs: 0%| | 0/50 [12:10<?, ?it/s]
train: 0%| | 0/204 [00:00<?, ?it/s]
Traceback (most recent call last):
File "train_rcnn.py", line 276, in
lr_scheduler_each_iter = (cfg.TRAIN.OPTIMIZER == 'adam_onecycle')
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../tools/train_utils/train_utils.py", line 199, in train
loss, tb_dict, disp_dict = self._train_it(batch)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../tools/train_utils/train_utils.py", line 132, in _train_it
loss, tb_dict, disp_dict = self.model_fn(self.model, batch)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/train_functions.py", line 68, in model_fn
ret_dict = model(input_data)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, kwargs)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], *kwargs[0])
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/point_rcnn.py", line 53, in forward
rpn_output = self.rpn(input_data)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/rpn.py", line 126, in forward
backbone_xyz, backbone_features, img_feature, l_xy_cor = self.backbone_net(pts_input, img_input, xy_input) # (B, N, 3), (B, C, N)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, *kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 341, in forward
image = self.Cross_Fusion[i](li_features, first_img_gather_feature, li_xy_cor, image)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 116, in forward
point_features = self.P2IA_Layer(img_features, point_features)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, *kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 91, in forward
ri = self.fc1(img_feas_f)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, **kwargs)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/functional.py", line 1369, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)
/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/config.py:250: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
yaml_cfg = edict(yaml.load(f))
Hi, when I try to reproduce the code I get an error, my CUDA version is 10.0 and torch version is 1.2.0. can someone please tell me how I should go about fixing this error. Thanks.
/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/config.py:250: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f)) 1.0 1.0 2023-11-04 13:34:47,355 INFO **Start logging** 2023-11-04 13:34:47,355 INFO CUDA_VISIBLE_DEVICES=ALL 2023-11-04 13:34:47,355 INFO cfg_file cfgs/CAR_EPNet_plus_plus.yaml 2023-11-04 13:34:47,355 INFO train_mode rcnn_online 2023-11-04 13:34:47,356 INFO batch_size 16 2023-11-04 13:34:47,356 INFO epochs 50 2023-11-04 13:34:47,356 INFO workers 8 2023-11-04 13:34:47,356 INFO ckpt_save_interval 1 2023-11-04 13:34:47,356 INFO output_dir ./log/CAR_EPNet_plus_plus/ 2023-11-04 13:34:47,356 INFO mgpus True 2023-11-04 13:34:47,356 INFO data_path ../../data/ 2023-11-04 13:34:47,356 INFO ckpt None 2023-11-04 13:34:47,356 INFO rpn_ckpt None 2023-11-04 13:34:47,356 INFO gt_database None 2023-11-04 13:34:47,356 INFO rcnn_training_roi_dir None 2023-11-04 13:34:47,356 INFO rcnn_training_feature_dir None 2023-11-04 13:34:47,356 INFO train_with_eval False 2023-11-04 13:34:47,356 INFO rcnn_eval_roi_dir None 2023-11-04 13:34:47,356 INFO rcnn_eval_feature_dir None 2023-11-04 13:34:47,356 INFO set_cfgs ['LI_FUSION.ENABLED', 'True', 'LI_FUSION.ADD_Image_Attention', 'True', 'CROSS_FUSION', 'True', 'USE_P2I_GATE', 'True', 'DEEP_RCNN_FUSION', 'False', 'USE_IMAGE_LOSS', 'True', 'IMAGE_WEIGHT', '1.0', 'USE_IMAGE_SCORE', 'True', 'USE_IMG_DENSE_LOSS', 'True', 'USE_MC_LOSS', 'True', 'MC_LOSS_WEIGHT', '1.0', 'I2P_Weight', '0.5', 'P2I_Weight', '0.5', 'ADD_MC_MASK', 'True', 'MC_MASK_THRES', '0.2', 'SAVE_MODEL_PREP', '0.8'] 2023-11-04 13:34:47,356 INFO model_type base 2023-11-04 13:34:47,356 INFO cfg.TAG: CAR_EPNet_plus_plus 2023-11-04 13:34:47,357 INFO cfg.CLASSES: Car 2023-11-04 13:34:47,357 INFO cfg.INCLUDE_SIMILAR_TYPE: True 2023-11-04 13:34:47,357 INFO cfg.AUG_DATA: True 2023-11-04 13:34:47,357 INFO cfg.AUG_METHOD_LIST: ['rotation', 'scaling', 'flip'] 2023-11-04 13:34:47,357 INFO cfg.AUG_METHOD_PROB: [1.0, 1.0, 0.5] 2023-11-04 13:34:47,357 INFO cfg.AUG_ROT_RANGE: 18 2023-11-04 13:34:47,357 INFO cfg.GT_AUG_ENABLED: False 2023-11-04 13:34:47,357 INFO cfg.GT_EXTRA_NUM: 15 2023-11-04 13:34:47,357 INFO cfg.GT_AUG_RAND_NUM: True 2023-11-04 13:34:47,357 INFO cfg.GT_AUG_APPLY_PROB: 1.0 2023-11-04 13:34:47,357 INFO cfg.GT_AUG_HARD_RATIO: 0.6 2023-11-04 13:34:47,357 INFO cfg.PC_REDUCE_BY_RANGE: True 2023-11-04 13:34:47,357 INFO cfg.PC_AREA_SCOPE: [[-40. 40. ] [ -1. 3. ] [ 0. 70.4]] 2023-11-04 13:34:47,358 INFO cfg.CLS_MEAN_SIZE: [[1.5256319 1.6285675 3.8831165]] 2023-11-04 13:34:47,358 INFO cfg.USE_IOU_BRANCH: True 2023-11-04 13:34:47,358 INFO cfg.USE_IM_DEPTH: False 2023-11-04 13:34:47,358 INFO cfg.USE_PSEUDO_LIDAR: False 2023-11-04 13:34:47,358 INFO cfg.CROSS_FUSION: True 2023-11-04 13:34:47,358 INFO cfg.INPUT_CROSS_FUSION: False 2023-11-04 13:34:47,358 INFO cfg.USE_KNN_FUSION: False 2023-11-04 13:34:47,358 INFO cfg.USE_SELF_ATTENTION: False 2023-11-04 13:34:47,358 INFO cfg.DEEP_RCNN_FUSION: False 2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_LOSS: True 2023-11-04 13:34:47,358 INFO cfg.IMAGE_WEIGHT: 1.0 2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_LOSS_TYPE: CrossEntropyLoss 2023-11-04 13:34:47,358 INFO cfg.USE_IMAGE_SCORE: True 2023-11-04 13:34:47,358 INFO cfg.USE_IMG_DENSE_LOSS: True 2023-11-04 13:34:47,358 INFO cfg.USE_KL_LOSS: False 2023-11-04 13:34:47,358 INFO cfg.USE_KL_LOSS_TYPE: KL 2023-11-04 13:34:47,358 INFO cfg.MC_LOSS_WEIGHT: 1.0 2023-11-04 13:34:47,359 INFO cfg.SAVE_MODEL_PREP: 0.8 2023-11-04 13:34:47,359 INFO cfg.USE_P2I_GATE: True 2023-11-04 13:34:47,359 INFO cfg.STACK_CROSS_FUSION: False 2023-11-04 13:34:47,359 INFO cfg.USE_IMAGE_RES: False 2023-11-04 13:34:47,359 INFO cfg.RCNN_IMG_CHANNEL: 32 2023-11-04 13:34:47,359 INFO cfg.ONLY_USE_IMAGE_FEAT: False 2023-11-04 13:34:47,359 INFO cfg.USE_POINT_ATT_FEATURE: False 2023-11-04 13:34:47,359 INFO cfg.USE_POINT_FEATURE_RES: False 2023-11-04 13:34:47,359 INFO cfg.I2P_Weight: 0.5 2023-11-04 13:34:47,359 INFO cfg.P2I_Weight: 0.5 2023-11-04 13:34:47,359 INFO cfg.USE_MC_LOSS: True 2023-11-04 13:34:47,359 INFO cfg.ADD_MC_MASK: True 2023-11-04 13:34:47,359 INFO cfg.MC_MASK_THRES: 0.2 2023-11-04 13:34:47,359 INFO cfg.USE_PURE_IMG_BACKBONE: False 2023-11-04 13:34:47,359 INFO cfg.USE_PAINTING_SCORE: False 2023-11-04 13:34:47,359 INFO cfg.USE_PAINTING_FEAT: False 2023-11-04 13:34:47,359 INFO
cfg.LI_FUSION = edict() 2023-11-04 13:34:47,359 INFO cfg.LI_FUSION.ENABLED: True 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.IMG_FEATURES_CHANNEL: 128 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.ADD_Image_Attention: True 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.IMG_CHANNELS: [3, 64, 128, 256, 512] 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.POINT_CHANNELS: [96, 256, 512, 1024] 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Reduce: [16, 16, 16, 16] 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Kernels: [2, 4, 8, 16] 2023-11-04 13:34:47,360 INFO cfg.LI_FUSION.DeConv_Strides: [2, 4, 8, 16] 2023-11-04 13:34:47,360 INFO
cfg.RPN = edict() 2023-11-04 13:34:47,360 INFO cfg.RPN.ENABLED: True 2023-11-04 13:34:47,360 INFO cfg.RPN.FIXED: False 2023-11-04 13:34:47,360 INFO cfg.RPN.USE_INTENSITY: False 2023-11-04 13:34:47,360 INFO cfg.RPN.USE_RGB: False 2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_XZ_FINE: True 2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_SCOPE: 3.0 2023-11-04 13:34:47,360 INFO cfg.RPN.LOC_BIN_SIZE: 0.5 2023-11-04 13:34:47,360 INFO cfg.RPN.NUM_HEAD_BIN: 12 2023-11-04 13:34:47,360 INFO cfg.RPN.BACKBONE: pointnet2_msg 2023-11-04 13:34:47,361 INFO cfg.RPN.USE_BN: True 2023-11-04 13:34:47,361 INFO cfg.RPN.NUM_POINTS: 16384 2023-11-04 13:34:47,361 INFO
cfg.RPN.SA_CONFIG = edict() 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.ATTN_DIM: 128 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.ATTN: [0, 0, 128, 128] 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.NPOINTS: [4096, 1024, 256, 64] 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.RADIUS: [[0.1, 0.5], [0.5, 1.0], [1.0, 2.0], [2.0, 4.0]] 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.NSAMPLE: [[16, 32], [16, 32], [16, 32], [16, 32]] 2023-11-04 13:34:47,361 INFO cfg.RPN.SA_CONFIG.MLPS: [[[16, 16, 32], [32, 32, 64]], [[64, 64, 128], [64, 96, 128]], [[128, 196, 256], [128, 196, 256]], [[256, 256, 512], [256, 384, 512]]] 2023-11-04 13:34:47,361 INFO cfg.RPN.FP_MLPS: [[128, 128], [256, 256], [512, 512], [512, 512]] 2023-11-04 13:34:47,361 INFO cfg.RPN.CLS_FC: [128] 2023-11-04 13:34:47,361 INFO cfg.RPN.REG_FC: [128] 2023-11-04 13:34:47,361 INFO cfg.RPN.DP_RATIO: 0.5 2023-11-04 13:34:47,361 INFO cfg.RPN.LOSS_CLS: SigmoidFocalLoss 2023-11-04 13:34:47,361 INFO cfg.RPN.FG_WEIGHT: 15 2023-11-04 13:34:47,361 INFO cfg.RPN.FOCAL_ALPHA: [0.25, 0.75] 2023-11-04 13:34:47,361 INFO cfg.RPN.FOCAL_GAMMA: 2.0 2023-11-04 13:34:47,361 INFO cfg.RPN.REG_LOSS_WEIGHT: [1.0, 1.0, 1.0, 1.0] 2023-11-04 13:34:47,362 INFO cfg.RPN.LOSS_WEIGHT: [1.0, 1.0] 2023-11-04 13:34:47,362 INFO cfg.RPN.NMS_TYPE: normal 2023-11-04 13:34:47,362 INFO cfg.RPN.SCORE_THRESH: 0.2 2023-11-04 13:34:47,362 INFO
cfg.RCNN = edict() 2023-11-04 13:34:47,362 INFO cfg.RCNN.ENABLED: True 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_RPN_FEATURES: True 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_MASK: True 2023-11-04 13:34:47,362 INFO cfg.RCNN.MASK_TYPE: seg 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_INTENSITY: False 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_DEPTH: True 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_SEG_SCORE: False 2023-11-04 13:34:47,362 INFO cfg.RCNN.ROI_SAMPLE_JIT: True 2023-11-04 13:34:47,362 INFO cfg.RCNN.ROI_FG_AUG_TIMES: 10 2023-11-04 13:34:47,362 INFO cfg.RCNN.REG_AUG_METHOD: multiple 2023-11-04 13:34:47,362 INFO cfg.RCNN.POOL_EXTRA_WIDTH: 0.2 2023-11-04 13:34:47,362 INFO cfg.RCNN.USE_RGB: False 2023-11-04 13:34:47,362 INFO cfg.RCNN.LOC_SCOPE: 1.5 2023-11-04 13:34:47,362 INFO cfg.RCNN.LOC_BIN_SIZE: 0.5 2023-11-04 13:34:47,363 INFO cfg.RCNN.NUM_HEAD_BIN: 9 2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_BY_BIN: False 2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_SCOPE: 0.5 2023-11-04 13:34:47,363 INFO cfg.RCNN.LOC_Y_BIN_SIZE: 0.25 2023-11-04 13:34:47,363 INFO cfg.RCNN.SIZE_RES_ON_ROI: False 2023-11-04 13:34:47,363 INFO cfg.RCNN.USE_BN: False 2023-11-04 13:34:47,363 INFO cfg.RCNN.DP_RATIO: 0.0 2023-11-04 13:34:47,363 INFO cfg.RCNN.BACKBONE: pointnet 2023-11-04 13:34:47,363 INFO cfg.RCNN.XYZ_UP_LAYER: [128, 128] 2023-11-04 13:34:47,363 INFO cfg.RCNN.NUM_POINTS: 512 2023-11-04 13:34:47,363 INFO
cfg.RCNN.SA_CONFIG = edict() 2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.NPOINTS: [128, 32, -1] 2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.RADIUS: [0.2, 0.4, 100] 2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.NSAMPLE: [64, 64, 64] 2023-11-04 13:34:47,363 INFO cfg.RCNN.SA_CONFIG.MLPS: [[128, 128, 128], [128, 128, 256], [256, 256, 512]] 2023-11-04 13:34:47,363 INFO cfg.RCNN.CLS_FC: [512, 512] 2023-11-04 13:34:47,363 INFO cfg.RCNN.REG_FC: [512, 512] 2023-11-04 13:34:47,363 INFO cfg.RCNN.LOSS_CLS: BinaryCrossEntropy 2023-11-04 13:34:47,364 INFO cfg.RCNN.FOCAL_ALPHA: [0.25, 0.75] 2023-11-04 13:34:47,364 INFO cfg.RCNN.FOCAL_GAMMA: 2.0 2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_WEIGHT: [1. 1. 1.] 2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_FG_THRESH: 0.6 2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_BG_THRESH: 0.45 2023-11-04 13:34:47,364 INFO cfg.RCNN.CLS_BG_THRESH_LO: 0.05 2023-11-04 13:34:47,364 INFO cfg.RCNN.REG_FG_THRESH: 0.55 2023-11-04 13:34:47,364 INFO cfg.RCNN.FG_RATIO: 0.5 2023-11-04 13:34:47,364 INFO cfg.RCNN.ROI_PER_IMAGE: 64 2023-11-04 13:34:47,364 INFO cfg.RCNN.HARD_BG_RATIO: 0.8 2023-11-04 13:34:47,364 INFO cfg.RCNN.IOU_LOSS_TYPE: raw 2023-11-04 13:34:47,364 INFO cfg.RCNN.IOU_ANGLE_POWER: 1 2023-11-04 13:34:47,364 INFO cfg.RCNN.SCORE_THRESH: 0.2 2023-11-04 13:34:47,364 INFO cfg.RCNN.NMS_THRESH: 0.1 2023-11-04 13:34:47,364 INFO
cfg.TRAIN = edict() 2023-11-04 13:34:47,364 INFO cfg.TRAIN.SPLIT: train 2023-11-04 13:34:47,365 INFO cfg.TRAIN.VAL_SPLIT: smallval 2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR: 0.002 2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_CLIP: 1e-05 2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_DECAY: 0.5 2023-11-04 13:34:47,365 INFO cfg.TRAIN.DECAY_STEP_LIST: [100, 150, 180, 200] 2023-11-04 13:34:47,365 INFO cfg.TRAIN.LR_WARMUP: True 2023-11-04 13:34:47,365 INFO cfg.TRAIN.WARMUP_MIN: 0.0002 2023-11-04 13:34:47,365 INFO cfg.TRAIN.WARMUP_EPOCH: 1 2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_MOMENTUM: 0.1 2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_DECAY: 0.5 2023-11-04 13:34:47,365 INFO cfg.TRAIN.BNM_CLIP: 0.01 2023-11-04 13:34:47,365 INFO cfg.TRAIN.BN_DECAY_STEP_LIST: [1000] 2023-11-04 13:34:47,365 INFO cfg.TRAIN.OPTIMIZER: adam_onecycle 2023-11-04 13:34:47,365 INFO cfg.TRAIN.WEIGHT_DECAY: 0.001 2023-11-04 13:34:47,365 INFO cfg.TRAIN.MOMENTUM: 0.9 2023-11-04 13:34:47,365 INFO cfg.TRAIN.MOMS: [0.95, 0.85] 2023-11-04 13:34:47,365 INFO cfg.TRAIN.DIV_FACTOR: 10.0 2023-11-04 13:34:47,365 INFO cfg.TRAIN.PCT_START: 0.4 2023-11-04 13:34:47,366 INFO cfg.TRAIN.GRAD_NORM_CLIP: 1.0 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_PRE_NMS_TOP_N: 9000 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_POST_NMS_TOP_N: 512 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_NMS_THRESH: 0.85 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_DISTANCE_BASED_PROPOSE: True 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_TRAIN_WEIGHT: 1.0 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RCNN_TRAIN_WEIGHT: 1.0 2023-11-04 13:34:47,366 INFO cfg.TRAIN.CE_WEIGHT: 5.0 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RPN_CE_WEIGHT: 5.0 2023-11-04 13:34:47,366 INFO cfg.TRAIN.IOU_LOSS_TYPE: cls_mask_with_bin 2023-11-04 13:34:47,366 INFO cfg.TRAIN.BBOX_AVG_BY_BIN: True 2023-11-04 13:34:47,366 INFO cfg.TRAIN.RY_WITH_BIN: False 2023-11-04 13:34:47,366 INFO
cfg.TEST = edict() 2023-11-04 13:34:47,366 INFO cfg.TEST.SPLIT: val 2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_PRE_NMS_TOP_N: 9000 2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_POST_NMS_TOP_N: 100 2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_NMS_THRESH: 0.8 2023-11-04 13:34:47,366 INFO cfg.TEST.RPN_DISTANCE_BASED_PROPOSE: True 2023-11-04 13:34:47,366 INFO cfg.TEST.BBOX_AVG_BY_BIN: True 2023-11-04 13:34:47,366 INFO cfg.TEST.RY_WITH_BIN: False cp: -r not specified; omitting directory '../lib/' cp: -r not specified; omitting directory '../tools' cp: 无法获取'../*.py' 的文件状态(stat): 没有那个文件或目录 ./log/CAR_EPNet_plus_plus/ 2023-11-04 13:34:47,378 INFO Loading TRAIN samples from ../../data/KITTI/object/training/label_2 ... 2023-11-04 13:34:47,892 INFO Done: filter TRAIN results: 3265 / 3712
##############USE Fusion_Cross_Conv_Gate(ADD)######### ##############ADDITION PI2 ATTENTION######### ##############USE Fusion_Cross_Conv_Gate(ADD)######### ##############ADDITION PI2 ATTENTION######### ##############USE Fusion_Cross_Conv_Gate(ADD)######### ##############ADDITION PI2 ATTENTION######### ##############USE Fusion_Cross_Conv_Gate(ADD)######### ##############ADDITION PI2 ATTENTION######### 2023-11-04 13:42:55,602 INFO **Start training** epochs: 0%| | 0/50 [12:10<?, ?it/s] train: 0%| | 0/204 [00:00<?, ?it/s] Traceback (most recent call last):
lr_scheduler_each_iter = (cfg.TRAIN.OPTIMIZER == 'adam_onecycle')
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../tools/train_utils/train_utils.py", line 199, in train
loss, tb_dict, disp_dict = self._train_it(batch)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../tools/train_utils/train_utils.py", line 132, in _train_it
loss, tb_dict, disp_dict = self.model_fn(self.model, batch)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/train_functions.py", line 68, in model_fn
ret_dict = model(input_data)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, kwargs)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], *kwargs[0])
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/point_rcnn.py", line 53, in forward
rpn_output = self.rpn(input_data)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/rpn.py", line 126, in forward
backbone_xyz, backbone_features, img_feature, l_xy_cor = self.backbone_net(pts_input, img_input, xy_input) # (B, N, 3), (B, C, N)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, *kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 341, in forward
image = self.Cross_Fusion[i](li_features, first_img_gather_feature, li_xy_cor, image)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 116, in forward
point_features = self.P2IA_Layer(img_features, point_features)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(*input, *kwargs)
File "/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/net/pointnet2_msg.py", line 91, in forward
ri = self.fc1(img_feas_f)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call
result = self.forward(input, **kwargs)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 87, in forward
return F.linear(input, self.weight, self.bias)
File "/home/bitcqic/anaconda3/envs/EPNetV2/lib/python3.7/site-packages/torch/nn/functional.py", line 1369, in linear
ret = torch.addmm(bias, input, weight.t())
RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling
File "train_rcnn.py", line 276, in
cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)/home/bitcqic/EPNetV2/EPNetV2/tools/../lib/config.py:250: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details. yaml_cfg = edict(yaml.load(f))