Closed ran337287 closed 6 years ago
ruoqianguo
commented
6 years ago @ran337287 hi, my GPU is GeForce GTX 1080 Ti. Do you compile the cuda code? Is there an error when compiling cuda code? set '-arch=sm_35'. Can you show your cuda error information?
ruoqianguo
commented
6 years ago Sorry, i can't see your 'img', but i think you should set '-arch=sm_35' in make.sh and sh make.sh again. you can see that Matching SM architectures (CUDA arch and CUDA gencode) for various NVIDIA cards
ran337287
commented
6 years ago Called with args:
Namespace(batch_size=1, checkepoch=1, checkpoint=0, checkpoint_interval=10000, checksession=1, class_agnostic=False, cuda=True, dataset='pascal_voc', disp_interval=100, exp_name='exp_name2', lr=0.001, lr_decay_gamma=0.1, lr_decay_step=5, lscale=False, mGPUs=False, max_epochs=12, net='detnet59', num_workers=4, optimizer='sgd', resume=False, save_dir='weights', session=1, start_epoch=1, use_tfboard=True)
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [8, 16, 32],
'CROP_RESIZE_WITH_MAX_POOL': False,
'CUDA': False,
'DATA_DIR': '/home/lxq/xiaoqian_Program/icdar/ori_DetNet_pytorch/data',
'DEDUP_BOXES': 0.0625,
'DETNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'EPS': 1e-14,
'EXP_DIR': 'res101',
'FEAT_STRIDE': [16],
'FPN_ANCHOR_SCALES': [32, 64, 128, 256, 512],
'FPN_ANCHOR_STRIDE': 1,
'FPN_FEAT_STRIDES': [4, 8, 16, 16, 16],
'GPU_ID': 0,
'HAS_MASK': True,
'MATLAB': 'matlab',
'MAX_NUM_GT_BOXES': 20,
'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
'FIXED_LAYERS': 5,
'REGU_DEPTH': False,
'WEIGHT_DECAY': 4e-05},
'PIXEL_MEANS': array([[[ 0.485, 0.456, 0.406]]]),
'PIXEL_STDS': array([[[ 0.229, 0.224, 0.225]]]),
'POOLING_MODE': 'align',
'POOLING_SIZE': 14,
'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '/home/lxq/xiaoqian_Program/icdar/ori_DetNet_pytorch',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'MODE': 'nms',
'NMS': 0.3,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_CROPPING': True,
'ASPECT_GROUPING': False,
'BATCH_SIZE': 128,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'BN_TRAIN': False,
'DISPLAY': 20,
'DOUBLE_BIAS': False,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
'STEPSIZE': [30000],
'SUMMARY_INTERVAL': 180,
'TRIM_HEIGHT': 600,
'TRIM_WIDTH': 600,
'TRUNCATED': False,
'USE_ALL_GT': True,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0001},
'USE_GPU_NMS': True}
Loaded dataset voc_2007_trainval for training
Set proposal method: gt
Appending horizontally-flipped training examples...
/usr/lib/python2.7/dist-packages/scipy/sparse/coo.py:182: VisibleDeprecationWarning: rank is deprecated; use the ndim attribute or function instead. To find the rank of a matrix see numpy.linalg.matrix_rank.
if np.rank(M) != 2:
/usr/lib/python2.7/dist-packages/scipy/sparse/coo.py:200: VisibleDeprecationWarning: rank is deprecated; use the ndim attribute or function instead. To find the rank of a matrix see numpy.linalg.matrix_rank.
if np.rank(self.data) != 1 or np.rank(self.row) != 1 or np.rank(self.col) != 1:
/usr/lib/python2.7/dist-packages/scipy/sparse/compressed.py:130: VisibleDeprecationWarning: rank is deprecated; use the ndim attribute or function instead. To find the rank of a matrix see numpy.linalg.matrix_rank.
if np.rank(self.data) != 1 or np.rank(self.indices) != 1 or np.rank(self.indptr) != 1:
wrote gt roidb to /home/lxq/xiaoqian_Program/icdar/ori_DetNet_pytorch/data/cache/voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
before filtering, there are 10022 images...
after filtering, there are 10022 images...
10022 roidb entries
Loading pretrained weights from data/pretrained_model/detnet59.pth
THCudaCheck FAIL file=/pytorch/torch/lib/THC/THCTensorCopy.cu line=204 error=8 : invalid device function
Traceback (most recent call last):
File "/home/lxq/xiaoqian_Program/icdar/ori_DetNet_pytorch/trainval_net.py", line 362, in
I tested -arch=sm_35, but there is the same error.
ruoqianguo
commented
6 years ago @ran337287 you can see this issue, it's the same error
ran337287
commented
6 years ago Thank you very much. It is helpful for me to solve the error.
hello, is your GPU Titan 1080 Ti? I use a single GPU of K40，and bs is set as 1, and it'll produce cuda runtime error.