pkuCactus
commented
5 years ago I run it again and failed in the same 840 iterations
Closed pkuCactus closed 5 years ago
pkuCactus
commented
5 years ago I run it again and failed in the same 840 iterations
PkuRainBow
commented
4 years ago I also meet the same error. Could you share how to address this problem?
tonypeng
commented
4 years ago running into this as well @pkuCactus @PkuRainBow were you able to fix the problem?
Calsia
commented
3 years ago maybe you can see https://github.com/uber-research/UPSNet/issues/36. it works for me.
Hi, Has any one encounter the problem that
IndexError: Out of bounds on buffer access (axis 0)? I'm using the Horovod with 16 GPUs for training and after 840 training iters, the training procedure failed. The logs are attached below,bbox_loss=0.229750, mask_loss=0.375499, fcn_loss=0.201432, panoptic_accuracy=0.931710, panoptic_loss=0.195533, INFO:root:Batch [840] Speed: 19.59 samples/sec Train-rpn_cls_loss=0.051714, rpn_bbox_loss=0.148433, rcnn_accuracy=0.889802, cls_loss=0.285857, bbox_loss=0.228536, mask_loss=0.373881, fcn_loss=0.201655, panoptic_accuracy=0.931608, panoptic_loss=0.196187, 2019-09-03 14:38:38,282 | callback.py | line 40 : Batch [840] Speed: 19.59 samples/sec Train-rpn_cls_loss=0.051714, rpn_bbox_loss=0.148433, rcnn_accuracy=0.889802, cls_loss=0.285857, bbox_loss=0.228536, mask_loss=0.373881, fcn_loss=0.201655, panoptic_accuracy=0.931608, panoptic_loss=0.196187, /home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log targets_dw = ww np.log(gt_widths / ex_widths) /home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log targets_dw = ww np.log(gt_widths / ex_widths) /home/work/user-job-dir/UPSNet/upsnet/../upsnet/bbox/bbox_transform.py:358: RuntimeWarning: invalid value encountered in log targets_dw = ww np.log(gt_widths / ex_widths) /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] /home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py:229: RuntimeWarning: invalid value encountered in greater_equal keep = np.where((ws >= min_size) & (hs >= min_size))[0] Traceback (most recent call last): File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward( input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)
Traceback (most recent call last):
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 440, in
upsnet_train()
File "UPSNet/upsnet/upsnet_end2end_train_hvd.py", line 242, in upsnet_train
output = train_model(data, label)
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(*input, *kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/models/resnetupsnet.py", line 109, in forward
rois, = self.pyramid_proposal(rpn_cls_prob, rpn_bbox_pred, data['im_info'])
File "/home/work/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in call
result = self.forward(input, **kwargs)
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/modules/pyramid_proposal.py", line 58, in forward
bbox_pred[3][[i], :, :, :], bbox_pred[4][[i], :, :, :], torch.from_numpy(im_info[i, :]))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/operators/functions/pyramid_proposal.py", line 168, in forward
keep = nms(np.hstack((proposals, scores)).astype(np.float32))
File "/home/work/user-job-dir/UPSNet/upsnet/../upsnet/nms/nms.py", line 45, in _nms
return gpu_nms(dets, thresh, device_id)
File "gpu_nms.pyx", line 36, in gpu_nms.gpu_nms
IndexError: Out of bounds on buffer access (axis 0)