Index out of bound error

[caiPactera]

Hi, I got an index out of bound error when training my own model.

loading annotations into memory... Done (t=0.35s) creating index... index created! num_images 11500 COCO_train2014 gt roidb loaded from ./data/cache/COCO_train2014_gt_roidb.pkl appending ground truth annotations Reading cached proposals after *NMS*** from data/proposals/COCO_train2014_rpn_after_nms.pkl Done! append flipped images to roidb filtered 7928 roidb entries: 23000 -> 15072 add bounding box regression targets bbox target means: [[0. 0. 0. 0.] [0. 0. 0. 0.]] [0. 0. 0. 0.] bbox target stdevs: [[0.1 0.1 0.2 0.2] [0.1 0.1 0.2 0.2]] [0.1 0.1 0.2 0.2] Creating Iterator with 15072 Images Total number of extracted chips: 72201 Done! The Iterator has 72201 samples! Initializing the model... Optimizer params: {'wd': 0.01, 'lr_scheduler': <train_utils.lr_scheduler.WarmupMultiBatchScheduler object at 0x7fee7fce4610>, 'multi_precision': True, 'learning_rate': 0.00015, 'rescale_grad': 1.0, 'clip_gradient': None, 'momentum': 0.9} aaa [00:32:22] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable) Exception in thread Thread-8: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(self.__args, self.__kwargs) File "lib/iterators/PrefetchingIter.py", line 61, in prefetch_func self.next_batch[i] = self.iters[i].next() File "lib/iterators/MNIteratorBase.py", line 90, in next if self.iter_next(): File "lib/iterators/MNIteratorBase.py", line 87, in iter_next return self.get_batch() File "lib/iterators/MNIteratorE2E.py", line 99, in get_batch self.batch = self._get_batch() File "lib/iterators/MNIteratorE2E.py", line 136, in _get_batch nids = processed_roidb[i]['props_in_chips'][cropid] IndexError: index 12 is out of bounds for axis 0 with size 12

How can I fix this?

[zhuomuniaoWD]

@caiPactera @mahyarnajibi I meet the same problem, can you solved it ?

[caiPactera]

I changed setting, deleted the '--set TRAIN.USE_NEG_CHIPS False' param and used rpn output from another model as input. This time I got no error. However, my program stuck into

add bounding box regression targets bbox target means: [[0. 0. 0. 0.] [0. 0. 0. 0.]] [0. 0. 0. 0.] bbox target stdevs: [[0.1 0.1 0.2 0.2] [0.1 0.1 0.2 0.2]] [0.1 0.1 0.2 0.2] Creating Iterator with 15072 Images Total number of extracted chips: 102358 Done! The Iterator has 102358 samples! Initializing the model... Optimizer params: {'wd': 0.01, 'lr_scheduler': <train_utils.lr_scheduler.WarmupMultiBatchScheduler object at 0x7f2780252c10>, 'multi_precision': True, 'learning_rate': 0.00015, 'rescale_grad': 1.0, 'clip_gradient': None, 'momentum': 0.9} aaa [17:53:19] src/operator/nn/./cudnn/./cudnn_algoreg-inl.h:107: Running performance tests to find the best convolution algorithm, this can take a while... (setting env variable MXNET_CUDNN_AUTOTUNE_DEFAULT to 0 to disable)

for hours. Is it an error? And how can I fix this new problem?

[dingjiansw101]

@caiPactera @zhuomuniaoWD @mahyarnajibi I also meet the same problem, it seems that there is a bug in the line? https://github.com/mahyarnajibi/SNIPER/blob/e01af0958c521cf181944b03c47169b1847df15e/lib/iterators/MNIteratorE2E.py#L57 Does it only assign props of the first chip to ['props_in_chips']? I changed 'ps[0]' to 'ps'. Then the problem of "Index out of bound" disappeared, but also stuck like @caiPactera Sorry, it seems that I used the code where it did not change

return props_in_chips

to https://github.com/mahyarnajibi/SNIPER/blob/e01af0958c521cf181944b03c47169b1847df15e/lib/data_utils/data_workers.py#L478

[adityag6994]

for me , i changed this line in

sniper_res101_e2e.yml

: gpus: '0, 1'