Launch traing for ssd-300 and meet DegenerateBatchError

[momo1986]

If you open a GitHub issue, here is the policy:

Your issue must be about one of the following:

1. a bug,

I tried to launch a ssd-300 training network.

After one epoch, a problem happens:

Epoch 1/200

Epoch 00001: LearningRateScheduler setting learning rate to 0.001. 199/200 [============================>.] - ETA: 2s - loss: 3.4431Traceback (most recent call last): File "train_ssd300.py", line 215, in initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, *kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1415, in fit_generator initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 230, in fit_generator workers=0) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1469, in evaluate_generator verbose=verbose) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 327, in evaluate_generator generator_output = next(output_generator) File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get six.reraise(value.class, value, value.traceback) File "/root/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 658, in _data_generator_task generator_output = next(self._generator) File "/fast/junyan/HandDetection/ssd_keras/data_generator/object_detection_2d_data_generator.py", line 1291, in generate "must be homogenous in size along all axes.") data_generator.object_detection_2d_data_generator.DegenerateBatchError: You produced an empty batch. This might be because the images in the batch vary in their size and/or number of channels. Note that after all transformations (if any were given) have been applied to all images in the batch, all images must be homogenous in size along all axes. `Epoch 1/200

Epoch 00001: LearningRateScheduler setting learning rate to 0.001. 199/200 [============================>.] - ETA: 2s - loss: 3.4431Traceback (most recent call last): File "train_ssd300.py", line 215, in initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, *kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1415, in fit_generator initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 230, in fit_generator workers=0) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1469, in evaluate_generator verbose=verbose) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 327, in evaluate_generator generator_output = next(output_generator) File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get six.reraise(value.class, value, value.traceback) File "/root/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 658, in _data_generator_task generator_output = next(self._generator) File "/fast/junyan/HandDetection/ssd_keras/data_generator/object_detection_2d_data_generator.py", line 1291, in generate "must be homogenous in size along all axes.") data_generator.object_detection_2d_data_generator.DegenerateBatchError: You produced an empty batch. This might be because the images in the batch vary in their size and/or number of channels. Note that after all transformations (if any were given) have been applied to all images in the batch, all images must be homogenous in size along all axes.

`from keras.optimizers import Adam, SGD
from keras.callbacks import ModelCheckpoint, LearningRateScheduler, TerminateOnNaN, CSVLogger
from keras import backend as K
from keras.models import load_model
from math import ceil
import numpy as np
from matplotlib import pyplot as plt

from models.keras_ssd300 import ssd_300
from keras_loss_function.keras_ssd_loss import SSDLoss
from keras_layers.keras_layer_AnchorBoxes import AnchorBoxes
from keras_layers.keras_layer_DecodeDetections import DecodeDetections
from keras_layers.keras_layer_DecodeDetectionsFast import DecodeDetectionsFast
from keras_layers.keras_layer_L2Normalization import L2Normalization

from ssd_encoder_decoder.ssd_input_encoder import SSDInputEncoder
from ssd_encoder_decoder.ssd_output_decoder import decode_detections, decode_detections_fast

from data_generator.object_detection_2d_data_generator import DataGenerator
from data_generator.object_detection_2d_geometric_ops import Resize
from data_generator.object_detection_2d_photometric_ops import ConvertTo3Channels
from data_generator.data_augmentation_chain_original_ssd import SSDDataAugmentation
from data_generator.object_detection_2d_misc_utils import apply_inverse_transforms
img_height = 300 # Height of the model input images
img_width = 300 # Width of the model input images
img_channels = 3 # Number of color channels of the model input images
mean_color = [123, 117, 104] # The per-channel mean of the images in the dataset. Do not change this value if you're using any of the pre-trained weights.
swap_channels = [2, 1, 0] # The color channel order in the original SSD is BGR, so we'll have the model reverse the color channel order of the input images.
n_classes = 20 # Number of positive classes, e.g. 20 for Pascal VOC, 80 for MS COCO
scales_pascal = [0.1, 0.2, 0.37, 0.54, 0.71, 0.88, 1.05] # The anchor box scaling factors used in the original SSD300 for the Pascal VOC datasets
scales_coco = [0.07, 0.15, 0.33, 0.51, 0.69, 0.87, 1.05] # The anchor box scaling factors used in the original SSD300 for the MS COCO datasets
scales = scales_pascal
aspect_ratios = [[1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5, 3.0, 1.0/3.0],
                 [1.0, 2.0, 0.5],
                 [1.0, 2.0, 0.5]] # The anchor box aspect ratios used in the original SSD300; the order matters
two_boxes_for_ar1 = True
steps = [8, 16, 32, 64, 100, 300] # The space between two adjacent anchor box center points for each predictor layer.
offsets = [0.5, 0.5, 0.5, 0.5, 0.5, 0.5] # The offsets of the first anchor box center points from the top and left borders of the image as a fraction of the step size for each predictor layer.
clip_boxes = False # Whether or not to clip the anchor boxes to lie entirely within the image boundaries
variances = [0.1, 0.1, 0.2, 0.2] # The variances by which the encoded target coordinates are divided as in the original implementation
normalize_coords = True
# 1: Build the Keras model.
batch_size = 16

K.clear_session() # Clear previous models from memory.
train_dataset = DataGenerator(load_images_into_memory=False, hdf5_dataset_path=None)
validation_dataset = DataGenerator(load_images_into_memory=False, hdf5_dataset_path=None)
model = ssd_300(image_size=(img_height, img_width, img_channels),
                n_classes=n_classes,
                mode='training',
                l2_regularization=0.0005,
                scales=scales,
                aspect_ratios_per_layer=aspect_ratios,
                two_boxes_for_ar1=two_boxes_for_ar1,
                steps=steps,
                offsets=offsets,
                clip_boxes=clip_boxes,
                variances=variances,
                normalize_coords=normalize_coords,
                subtract_mean=mean_color,
                swap_channels=swap_channels)

# 2: Load some weights into the model.

# TODO: Set the path to the weights you want to load.
weights_path = './VGG_ILSVRC_16_layers_fc_reduced.h5'

model.load_weights(weights_path, by_name=True)

# 3: Instantiate an optimizer and the SSD loss function and compile the model.
#    If you want to follow the original Caffe implementation, use the preset SGD
#    optimizer, otherwise I'd recommend the commented-out Adam optimizer.

#adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, epsilon=1e-08, decay=0.0)
sgd = SGD(lr=0.001, momentum=0.9, decay=0.0, nesterov=False)

ssd_loss = SSDLoss(neg_pos_ratio=3, alpha=1.0)

model.compile(optimizer=sgd, loss=ssd_loss.compute_loss)
train_images_dir = '/fast/right_hand_images'
validation_images_dir = '/fast/right_hand_test'
# Ground truth
train_labels_filename = '/fast/right_hand_images/right_hand_label.csv'
validation_labels_filename = '/fast/right_hand_test/right_hand_test.csv'
train_dataset.parse_csv_center(images_dir=train_images_dir,
                        labels_filename=train_labels_filename,
                        input_format=['image_name', 'center_x', 'center_y', 'width', 'height', 'class_id'], # This is the order of the first six columns in the CSV file that contains the labels for your dataset. If your labels are in XML format, maybe the XML parser will be helpful, check the documentation.
                        include_classes='all')
validation_dataset.parse_csv_center(images_dir=validation_images_dir,
                        labels_filename=validation_labels_filename,
                        input_format=['image_name', 'center_x', 'center_y', 'width', 'height', 'class_id'], # This is the order of the first six columns in the CSV file that contains the labels for your dataset. If your labels are in XML format, maybe the XML parser will be helpful, check the documentation.
                        include_classes='all')

# Optional: Convert the dataset into an HDF5 dataset. This will require more disk space, but will
# speed up the training. Doing this is not relevant in case you activated the `load_images_into_memory`
# option in the constructor, because in that cas the images are in memory already anyway. If you don't
# want to create HDF5 datasets, comment out the subsequent two function calls.

train_dataset.create_hdf5_dataset(file_path='dataset_right_hand_train.h5',
                                  resize=False,
                                  variable_image_size=True,
                                  verbose=True)

validation_dataset.create_hdf5_dataset(file_path='dataset_right_hand_val.h5',
                                resize=False,
                                variable_image_size=True,
                                verbose=True)
ssd_data_augmentation = SSDDataAugmentation(img_height=img_height,
                                            img_width=img_width,
                                            background=mean_color)

# For the validation generator:

convert_to_3_channels = ConvertTo3Channels()
resize = Resize(height=img_height, width=img_width)

# 5: Instantiate an encoder that can encode ground truth labels into the format needed by the SSD loss function.

# The encoder constructor needs the spatial dimensions of the model's predictor layers to create the anchor boxes.
predictor_sizes = [model.get_layer('conv4_3_norm_mbox_conf').output_shape[1:3],
                   model.get_layer('fc7_mbox_conf').output_shape[1:3],
                   model.get_layer('conv6_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv7_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv8_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv9_2_mbox_conf').output_shape[1:3]]

ssd_input_encoder = SSDInputEncoder(img_height=img_height,
                                    img_width=img_width,
                                    n_classes=n_classes,
                                    predictor_sizes=predictor_sizes,
                                    scales=scales,
                                    aspect_ratios_per_layer=aspect_ratios,
                                    two_boxes_for_ar1=two_boxes_for_ar1,
                                    steps=steps,
                                    offsets=offsets,
                                    clip_boxes=clip_boxes,
                                    variances=variances,
                                    matching_type='multi',
                                    pos_iou_threshold=0.5,
                                    neg_iou_limit=0.5,
                                    normalize_coords=normalize_coords)

train_generator = train_dataset.generate(batch_size=batch_size,
                                         shuffle=True,
                                         transformations=[ssd_data_augmentation],
                                         label_encoder=ssd_input_encoder,
                                         returns={'processed_images',
                                                  'encoded_labels'},
                                         keep_images_without_gt=False)

val_generator = validation_dataset.generate(batch_size=batch_size,
                                     shuffle=False,
                                     transformations=[convert_to_3_channels,
                                                      resize],
                                     label_encoder=ssd_input_encoder,
                                     returns={'processed_images',
                                              'encoded_labels'},
                                     keep_images_without_gt=False)
# Define a learning rate schedule.
train_dataset_size = train_dataset.get_dataset_size()
val_dataset_size   = validation_dataset.get_dataset_size()

def lr_schedule(epoch):
    if epoch < 80:
        return 0.001
    elif epoch < 100:
        return 0.0001
    else:
        return 0.00001

# Define model callbacks.

# TODO: Set the filepath under which you want to save the model.
model_checkpoint = ModelCheckpoint(filepath='ssd300_pascal_07+12_epoch-{epoch:02d}_loss-{loss:.4f}_val_loss-{val_loss:.4f}.h5',
                                   monitor='val_loss',
                                   verbose=1,
                                   save_best_only=True,
                                   save_weights_only=False,
                                   mode='auto',
                                   period=1)
#model_checkpoint.best =

csv_logger = CSVLogger(filename='ssd300_pascal_07+12_training_log.csv',
                       separator=',',
                       append=True)

learning_rate_scheduler = LearningRateScheduler(schedule=lr_schedule,
                                                verbose=1)

terminate_on_nan = TerminateOnNaN()

callbacks = [model_checkpoint,
             csv_logger,
             learning_rate_scheduler,
             terminate_on_nan]

# If you're resuming a previous training, set `initial_epoch` and `final_epoch` accordingly.
initial_epoch   = 0
final_epoch     = 200
steps_per_epoch = 200

history = model.fit_generator(generator=train_generator,
                              steps_per_epoch=steps_per_epoch,
                              epochs=final_epoch,
                              callbacks=callbacks,
                              validation_data=val_generator,
                              validation_steps=val_dataset_size//batch_size,
                              initial_epoch=initial_epoch)

` My Input data is center-point and width, height,

I do transfer like this:

    def parse_csv_center(self,
                  images_dir,
                  labels_filename,
                  input_format,
                  include_classes='all',
                  random_sample=False,
                  ret=False,
                  verbose=True,
                  img_width= 1920,
                  img_height=1080):
        '''
        Arguments:
            images_dir (str): The path to the directory that contains the images.
            labels_filename (str): The filepath to a CSV file that contains one ground truth bounding box per line
                and each line contains the following six items: image file name, class ID, xmin, xmax, ymin, ymax.
                The six items do not have to be in a specific order, but they must be the first six columns of
                each line. The order of these items in the CSV file must be specified in `input_format`.
                The class ID is an integer greater than zero. Class ID 0 is reserved for the background class.
                `xmin` and `xmax` are the left-most and right-most absolute horizontal coordinates of the box,
                `ymin` and `ymax` are the top-most and bottom-most absolute vertical coordinates of the box.
                The image name is expected to be just the name of the image file without the directory path
                at which the image is located.
            input_format (list): A list of six strings representing the order of the six items
                image file name, class ID, center_x, center_y, width, height in the input CSV file. The expected strings
                are 'image_name', 'center_x', 'center_y', 'width', 'height', 'class_id'.
            include_classes (list, optional): Either 'all' or a list of integers containing the class IDs that
                are to be included in the dataset. If 'all', all ground truth boxes will be included in the dataset.
            random_sample (float, optional): Either `False` or a float in `[0,1]`. If this is `False`, the
                full dataset will be used by the generator. If this is a float in `[0,1]`, a randomly sampled
                fraction of the dataset will be used, where `random_sample` is the fraction of the dataset
                to be used. For example, if `random_sample = 0.2`, 20 precent of the dataset will be randomly selected,
                the rest will be ommitted. The fraction refers to the number of images, not to the number
                of boxes, i.e. each image that will be added to the dataset will always be added with all
                of its boxes.
            ret (bool, optional): Whether or not to return the outputs of the parser.
            verbose (bool, optional): If `True`, prints out the progress for operations that may take a bit longer.

        Returns:
            None by default, optionally lists for whichever are available of images, image filenames, labels, and image IDs.
        '''

        # Set class members.
        self.images_dir = images_dir
        self.labels_filename = labels_filename
        self.input_format = input_format
        self.include_classes = include_classes

        # Before we begin, make sure that we have a labels_filename and an input_format
        if self.labels_filename is None or self.input_format is None:
            raise ValueError(
                "`labels_filename` and/or `input_format` have not been set yet. You need to pass them as arguments.")

        # Erase data that might have been parsed before
        self.filenames = []
        self.image_ids = []
        self.labels = []

        # First, just read in the CSV file lines and sort them.

        data = []

        with open(self.labels_filename, newline='') as csvfile:
            csvread = csv.reader(csvfile, delimiter=',')
            next(csvread)  # Skip the header row.
            for row in csvread:  # For every line (i.e for every bounding box) in the CSV file...
                if self.include_classes == 'all' or int(row[self.input_format.index(
                        'class_id')].strip()) in self.include_classes:  # If the class_id is among the classes that are to be included in the dataset...
                    box = []  # Store the box class and coordinates here
                    box.append(row[self.input_format.index(
                        'image_name')].strip())  # Select the image name column in the input format and append its content to `box`
                    x_min = int(float(row[self.input_format.index(
                        'center_x')] )-float( row[self.input_format.index('width')])*img_width/2)
                    x_max = int(float(row[self.input_format.index(
                        'center_x')]) + float(row[self.input_format.index('width')])*img_width/2)
                    y_min = int(float(row[self.input_format.index(
                        'center_y')]) - float(row[self.input_format.index('height')])*img_height/2)
                    y_max = int(float(row[self.input_format.index(
                        'center_y')]) +  float(row[self.input_format.index('height')])*img_height/2)
                    box.append(x_min)
                    box.append(x_max)
                    box.append(y_min)
                    box.append(y_max)
                    box.append(int(row[self.input_format.index('class_id')]))
                    data.append(box)

        data = sorted(data)  # The data needs to be sorted, otherwise the next step won't give the correct result

        # Now that we've made sure that the data is sorted by file names,
        # we can compile the actual samples and labels lists

        current_file = data[0][0]  # The current image for which we're collecting the ground truth boxes
        current_image_id = data[0][0].split('.')[
            0]  # The image ID will be the portion of the image name before the first dot.
        current_labels = []  # The list where we collect all ground truth boxes for a given image
        add_to_dataset = False
        for i, box in enumerate(data):

            if box[0] == current_file:  # If this box (i.e. this line of the CSV file) belongs to the current image file
                current_labels.append(box[1:])
                if i == len(data) - 1:  # If this is the last line of the CSV file
                    if random_sample:  # In case we're not using the full dataset, but a random sample of it.
                        p = np.random.uniform(0, 1)
                        if p >= (1 - random_sample):
                            self.labels.append(np.stack(current_labels, axis=0))
                            self.filenames.append(os.path.join(self.images_dir, current_file))
                            self.image_ids.append(current_image_id)
                    else:
                        self.labels.append(np.stack(current_labels, axis=0))
                        self.filenames.append(os.path.join(self.images_dir, current_file))
                        self.image_ids.append(current_image_id)
            else:  # If this box belongs to a new image file
                if random_sample:  # In case we're not using the full dataset, but a random sample of it.
                    p = np.random.uniform(0, 1)
                    if p >= (1 - random_sample):
                        self.labels.append(np.stack(current_labels, axis=0))
                        self.filenames.append(os.path.join(self.images_dir, current_file))
                        self.image_ids.append(current_image_id)
                else:
                    self.labels.append(np.stack(current_labels, axis=0))
                    self.filenames.append(os.path.join(self.images_dir, current_file))
                    self.image_ids.append(current_image_id)
                current_labels = []  # Reset the labels list because this is a new file.
                current_file = box[0]
                current_image_id = box[0].split('.')[0]
                current_labels.append(box[1:])
                if i == len(data) - 1:  # If this is the last line of the CSV file
                    if random_sample:  # In case we're not using the full dataset, but a random sample of it.
                        p = np.random.uniform(0, 1)
                        if p >= (1 - random_sample):
                            self.labels.append(np.stack(current_labels, axis=0))
                            self.filenames.append(os.path.join(self.images_dir, current_file))
                            self.image_ids.append(current_image_id)
                    else:
                        self.labels.append(np.stack(current_labels, axis=0))
                        self.filenames.append(os.path.join(self.images_dir, current_file))
                        self.image_ids.append(current_image_id)

        self.dataset_size = len(self.filenames)
        self.dataset_indices = np.arange(self.dataset_size, dtype=np.int32)
        if self.load_images_into_memory:
            self.images = []
            if verbose:
                it = tqdm(self.filenames, desc='Loading images into memory', file=sys.stdout)
            else:
                it = self.filenames
            for filename in it:
                with Image.open(filename) as image:
                    self.images.append(np.array(image, dtype=np.uint8))

        if ret:  # In case we want to return these
            return self.images, self.filenames, self.labels, self.image_ids

[momo1986]

I tried to make the validation data and training data with same format:

predictor_sizes = [model.get_layer('conv4_3_norm_mbox_conf').output_shape[1:3],
                   model.get_layer('fc7_mbox_conf').output_shape[1:3],
                   model.get_layer('conv6_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv7_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv8_2_mbox_conf').output_shape[1:3],
                   model.get_layer('conv9_2_mbox_conf').output_shape[1:3]]

ssd_input_encoder = SSDInputEncoder(img_height=img_height,
                                    img_width=img_width,
                                    n_classes=n_classes,
                                    predictor_sizes=predictor_sizes,
                                    scales=scales,
                                    aspect_ratios_per_layer=aspect_ratios,
                                    two_boxes_for_ar1=two_boxes_for_ar1,
                                    steps=steps,
                                    offsets=offsets,
                                    clip_boxes=clip_boxes,
                                    variances=variances,
                                    matching_type='multi',
                                    pos_iou_threshold=0.5,
                                    neg_iou_limit=0.5,
                                    normalize_coords=normalize_coords)

train_generator = train_dataset.generate(batch_size=batch_size,
                                         shuffle=True,
                                         transformations=[ssd_data_augmentation],
                                         label_encoder=ssd_input_encoder,
                                         returns={'processed_images',
                                                  'encoded_labels'},
                                         keep_images_without_gt=False)

val_generator = validation_dataset.generate(batch_size=batch_size,
                                     shuffle=False,
                                     transformations=[ssd_data_augmentation],
                                     label_encoder=ssd_input_encoder,
                                     returns={'processed_images',
                                              'encoded_labels'},
                                     keep_images_without_gt=False)
# Define a learning rate schedule.
train_dataset_size = train_dataset.get_dataset_size()
val_dataset_size   = validation_dataset.get_dataset_size()

However, still got such error: Epoch 1/200

Epoch 00001: LearningRateScheduler setting learning rate to 0.001. 19/20 [===========================>..] - ETA: 3s - loss: 2.9068Traceback (most recent call last): File "train_ssd300.py", line 214, in initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(*args, *kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1415, in fit_generator initial_epoch=initial_epoch) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 230, in fit_generator workers=0) File "/root/anaconda3/lib/python3.6/site-packages/keras/legacy/interfaces.py", line 91, in wrapper return func(args, **kwargs) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training.py", line 1469, in evaluate_generator verbose=verbose) File "/root/anaconda3/lib/python3.6/site-packages/keras/engine/training_generator.py", line 327, in evaluate_generator generator_output = next(output_generator) File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 793, in get six.reraise(value.class, value, value.traceback) File "/root/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/root/anaconda3/lib/python3.6/site-packages/keras/utils/data_utils.py", line 658, in _data_generator_task generator_output = next(self._generator) File "/fast/junyan/HandDetection/ssd_keras/data_generator/object_detection_2d_data_generator.py", line 1225, in generate batch_X[i], batch_y[i] = transform(batch_X[i], batch_y[i]) File "/fast/junyan/HandDetection/ssd_keras/data_generator/data_augmentation_chain_original_ssd.py", line 275, in call image, labels = transform(image, labels) File "/fast/junyan/HandDetection/ssd_keras/data_generator/data_augmentation_chain_original_ssd.py", line 144, in call return self.expand(image, labels, return_inverter) File "/fast/junyan/HandDetection/ssd_keras/data_generator/object_detection_2d_patch_sampling_ops.py", line 537, in call return self.sample_patch(image, labels, return_inverter) File "/fast/junyan/HandDetection/ssd_keras/data_generator/object_detection_2d_patch_sampling_ops.py", line 296, in call canvas[-patch_ymin:-patch_ymin + image_crop_height, -patch_xmin:-patch_xmin + image_crop_width] = image[:image_crop_height, :image_crop_width] ValueError: could not broadcast input array from shape (864,955,3) into shape (0,955,3)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dsnsabari]

I am getting the same error, could you tell me how to fix it?

[artur99]

I am getting the same error, could you tell me how to fix it?

Are you sure you're sending the parameters for training in the correct order (xmin, ymin, xmax, ymax)?

[dsnsabari]

yes. I cross-checked the values. When I debugged the code, I can some values in batch_items_to_remove. Could you tell me why I am getting empty batch_y?