Error when training custom dataset (out of range?)

[Elizabethcase]

Hi,

I'm running the training on a custom dataset whose images are 512x512 (using vgg_300 right now for convenience).

Getting an error every time I try to run train_ssd_network -- I think it has to do with a preprocessing step. Maybe this happens when a photo has no bounding boxes?

This happens whether or not I use the last 5 flags.

Thanks in advance.

NFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, indices[0] = 0 is not in [0, 0)
         [[Node: ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter/boolean_mask/Gather = Gather[Tindices=DT_INT64, Tparams=DT_INT64, validate_indices=true,
 _device="/job:localhost/replica:0/task:0/cpu:0"](ssd_preprocessing_train/distorted_bounding_box_crop/bboxes_filter/boolean_mask/Reshape, ssd_preprocessing_train/distorted
_bounding_box_crop/bboxes_filter/boolean_mask/Squeeze)]]
INFO:tensorflow:Caught OutOfRangeError. Stopping Training.

python CUSTOM_train_ssd_network.py \
--train_dir=${TRAIN_DIR} \
--dataset_dir=${DATASET_DIR} \
--dataset_name=cropped_reshaped_wks1_2 \
--dataset_split_name=train \
--model_name=ssd_300_vgg \
--checkpoint_path=${CHECKPOINT_PATH} \
--save_summaries_secs=60 \
--save_interval_secs=600 \
--weight_decay=0.0005 \
--optimizer=adam \
--learning_rate=0.001 \
--batch_size=4 \
--num_classes=13 \
--train_image_size=512 \
--checkpoint_exclude_scopes=ssd_300_vgg/conv6,ssd_300_vgg/conv7,ssd_300_vgg/block8,ssd_300_vgg/block9,ssd_300_vgg/block10,ssd_300_vgg/block11,ssd_300_vgg/block4_box,ssd_300_vgg/block7_box,ssd_300_vgg/block8_box,ssd_300_vgg/block9_box,ssd_300_vgg/block10_box,ssd_300_vgg/block11_box \
--trainable_scopes=ssd_300_vgg/conv6,ssd_300_vgg/conv7,ssd_300_vgg/block8,ssd_300_vgg/block9,ssd_300_vgg/block10,ssd_300_vgg/block11,ssd_300_vgg/block4_box,ssd_300_vgg/block7_box,ssd_300_vgg/block8_box,ssd_300_vgg/block9_box,ssd_300_vgg/block10_box,ssd_300_vgg/block11_box

[xiaoluojun]

HI，I'm meet this problem that is same with you.How are you deal with?

[RicklessMorty]

try decreasing the anchor size

[Cuky88]

I got the same problem with a custom dataset. Did you figure out how to solve this?

[sunrise666]

I got the same problem with a custom dataset. Did you figure out how to solve this?

[Backalla]

I encountered the similar problem. It turns out that custom dataset has some boxes on the edge of the image and while converting the box coordinates to range [0,1] it is going slightly above 1. I changed the pascalvoc_to_tfrecords.py file line 117 to bboxes.append((max(float(bbox.find('ymin').text) / shape[0], 0.0), max(float(bbox.find('xmin').text) / shape[1], 0.0), min(float(bbox.find('ymax').text) / shape[0], 1.0), min(float(bbox.find('xmax').text) / shape[1], 1.0) ))