How to train this on a new dataset ?

[suvojit-0x55aa]

I'm using a custom dataset. But it's asking for RPN weights, is there any way to bypass that. If I load the weights, it failing with the following error:

Loading data/proposals/voc_2007_trainval_rpn.pkl....
Done!
Applying NMS...
Traceback (most recent call last):
  File "main_train.py", line 59, in <module>
    for image_set in image_sets]
  File "lib/data_utils/load_data.py", line 32, in load_proposal_roidb
    roidb = eval('imdb.' + proposal + '_roidb')(roidb, append_gt,proposal_path=proposal_path)
  File "lib/dataset/imdb.py", line 139, in rpn_roidb
    rpn_roidb = self.load_rpn_roidb(gt_roidb,proposal_path)
  File "lib/dataset/imdb.py", line 126, in load_rpn_roidb
    box_list, mapping_list = self.load_rpn_data(proposal_path)
  File "lib/dataset/imdb.py", line 109, in load_rpn_data
    keeps = p.map(nmsp, ttboxes)
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 251, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/usr/lib/python2.7/multiprocessing/pool.py", line 567, in get
    raise self._value
IndexError: too many indices for array

[bharatsingh430]

this will be resolved soon, we are aware of the issue.

[bharatsingh430]

In the meantime you can use the openimages branch if you don’t have many small objects but it doesn’t include negative sampling. Images are expected to be around 1000x768 so only downsampling is done.

[Ram-Godavarthi]

@bharatsingh430 If i need to train on my own dataset of 15000 images. What changes should i do to detect 2 different classes? Is this repo compatible with pascal format dataset? Where am i supposed to give the batch size for training? I have many images, so i need larger batch size. so. I am using aws p2.xlarge instance. So what is the appropriate batch size??

[suvojit-0x55aa]

@bharatsingh430 thanks for the clarification. I've a dataset of high resolution images but the objects are small. Will using the openimages branch be effective ?

[bharatsingh430]

Yes, it will work (but little worse). the end to end version will be put out soon after some testing

[mahyarnajibi]

Hi @suvojit-0x55aa, Commit 8711340 addresses this issue by adding an option to the master branch which removes the need for proposals. Pull the SNIPER master branch and then you should be able to train on your new dataset by running python main_train.py --set TRAIN.USE_NEG_CHIPS False. In the near future, the option for training on new datasets with negative chip mining would be also added to the master. Hope it helps!

[xiaomengyc]

Hi @mahyarnajibi , I till got the error message AssertionError: rpn data not found at data/proposals/COCO_train2017_rpn.pkl using the command python main_train.py --set TRAIN.USE_NEG_CHIPS False.
Can you tell me how to generate this *_rpn.pkl file for a new dataset ?

Thanks a lot!

[mahyarnajibi]

Did you pull the master after commit 8711340? Just to make sure can you manually set the TRAIN.USE_NEG_CHIPS to False in configs/faster/default_configs.py. The script for training on new datasets with negative chip mining would be added in the next couple of days but right now you should be able to train without the need for the rpn data.

[xiaomengyc]

@mahyarnajibi Thank you! It works very well now!

[suvojit-0x55aa]

Thanks @mahyarnajibi @bharatsingh430 for the update. I'm trying to run the network with the new configs but i'm running into following error on a Tesla P100: CUDA error: too many resources requested for launch. Can you give me a pointer on what maybe happening ?

[ddt2014]

@suvojit-0x55aa hi have you solved this problem?

[suvojit-0x55aa]

@ddt2014 no I haven't

[pengyuan101]

have you added the code of training on new datasets with negative chip mining ?

Did you pull the master after commit 8711340? Just to make sure can you manually set the TRAIN.USE_NEG_CHIPS to False in configs/faster/default_configs.py. The script for training on new datasets with negative chip mining would be added in the next couple of days but right now you should be able to train without the need for the rpn data.