Problem with batch prediction

[Amilitar]

Got a problem with batch prediction, I tried to create a prediction for batch images and got the next problem.

• Create an instance for one prediction model
• Prepare images batch with size more then 1
• first prediction always successfully, but next prediction always return error tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape is a tensor with 148 values, but the requested shape requires a multiple of 8 [[node functional_1/tf_op_layer_Reshape_13/Reshape_13 (defined at /Documents/git/cinsay/tensorflow-yolov4-tflite/batch_prediction.py:54) ]] [Op:__inference_predict_function_11048]

I tried to do a batch experiment with saved model and with model which generated in save_model.py Interesting fact, if I generated a batch with random digits, the model worked And if I predict in batch different classes, the model worked, but when the batch contains, for example, the person then the model fallback

My test class ` from os import listdir from absl.flags import FLAGS

import cv2 import numpy as np import tensorflow as tf from PIL import Image from numpy import asarray, uint8, float32 from tensorflow.python.saved_model import tag_constants from tqdm import tqdm

class BatchPrediction(): def init(self, model) -> None: super().init() self.create_model(model)

def create_model(self, model):
    self.model = model

def prepare_images(self, images):
    images_metadata = []
    for file in images:
        # content = file.read()
        # cv2.imshow("fsd", file)
        # cv2.waitKey(0)
        # cv2.destroyAllWindows()
        # file = asarray(bytearray(file), dtype=uint8)
        # original_image = cv2.imdecode(file, cv2.IMREAD_COLOR)
        original_image = cv2.cvtColor(file, cv2.COLOR_BGR2RGB)
        image_data = cv2.resize(original_image, (FLAGS.input_size, FLAGS.input_size))
        image_data = image_data / 255.

        images_metadata.append(image_data)
    images_metadata = asarray(images_metadata).astype(float32)
    return images_metadata

def predict(self, images, category):
    images_data = self.prepare_images(images)
    random_image_data = np.random.random((3, FLAGS.input_size, FLAGS.input_size, 3))
    random_image_data = asarray(random_image_data).astype(float32)
    batch_data = tf.constant(images_data)
    # self.infer = self.saved_model_loaded.signatures['serving_default']

    # pred_bbox = self.infer(batch_data)
    return self.model.predict(batch_data)

@staticmethod
def get_files_from_directory(path, extension):
    files = listdir(path)
    file_names = [x for x in files if x.endswith(extension)]
    return file_names, len(files)

@staticmethod
def close_files(files):
    for file in files:
        file.close()

def start(self):
    folder = "path to image folder here"
    generator, file_count = self.get_files_from_directory(folder, ".jpg")
    files = []
    for file_name in tqdm(generator, total=file_count, desc="Proceed files"):
        # file = FileStorage(stream=open(file=folder + "/" + file_name, mode="rb", encoding=None),
        #                    content_type='image/jpeg')
        # image = Image.open(folder + "/" + file_name)
        file = cv2.imread(folder + "/" + file_name)

        # file = np.array(image.getdata())
        files.append(file)
        if len(files) == 2:
            result = self.predict(files, "person")
            print(result)
            # self.close_files(files)
            files = []

in save _model.py insert code batch_predict= BatchPrediction(model) batch_predict.start()before saving model

[aiswaryasukumar4]

@Amilitar @hunglc007 I am facing the same issue. Found any solution yet?

[Amilitar]

@Amilitar @hunglc007 I am facing the same issue. Found any solution yet?

Unfortunatly not( I found last worked commit, by 28.06.2020 8:26 (b8d17580) After that yolov4 was modified and batch process was broken.

Possible problem here pred_bbox = [tf.reshape(x, (tf.shape(x)[0], -1, tf.shape(x)[-1])) for x in bbox_tensors] pred_bbox = tf.concat(pred_bbox, axis=1) pred_prob = [tf.reshape(x, (tf.shape(x)[0], -1, tf.shape(x)[-1])) for x in prob_tensors] pred_prob = tf.concat(pred_prob, axis=1)

[pasin-k]

I also just found the same issue here. I tested some random things and found tIt seems that it can predict multiple times if you feed the exact same data (which is pointless but somehow it works) or if the next data and those after has a batch size of one.

Does anyone know if this is a problem with training as well? I'm currently using it for prediction but I will need to do some fine-tuning in the future,

Possible problem here pred_bbox = [tf.reshape(x, (tf.shape(x)[0], -1, tf.shape(x)[-1])) for x in bbox_tensors] pred_bbox = tf.concat(pred_bbox, axis=1) pred_prob = [tf.reshape(x, (tf.shape(x)[0], -1, tf.shape(x)[-1])) for x in prob_tensors] pred_prob = tf.concat(pred_prob, axis=1)

@Amilitar which file that contains the code you mentioned? I seem to cannot find one (at least for the current commit). Also, Is the commit you mentioned the last one that works with batch process?

Edit: I think the error comes from yolov4.filter_boxes on this line. class_boxes = tf.reshape(class_boxes, [tf.shape(scores)[0], -1, tf.shape(class_boxes)[-1]]) I'm going to check the source of the problem and update later if found the way to fix it.

[pasin-k]

@hunglc007 @Amilitar I believe that I found the current solution to the bug where there is error for the prediction of batch size > 1.

In the filter_boxes function, some of the prediction bbox got removed by some threshold and thus the dimension got reduced. And since data in each batch would have a different number of box filtered by this function, this would result in an unequal matrix size (and thus reshaping cause error). This works for a batch size of one though.

Here is my workaround. I only removed the bbox if all of the bbox in all batch (of the same bbox id) is below the threshold. This works the same for batch size of one and works normally for a batch size higher than one.

One disadvantage is that the number of bbox from the model can be much higher than before but the bbox got passed to tf.image.combined_non_max_suppression anyway so the final result is the same.

This could be optimized a little bit more though so filter_boxes give a smaller matrix but at least it works.

Here is the code I edited on yolov4.py. Added two lines to the code.

def filter_boxes(box_xywh, scores, score_threshold=0.4, input_shape=tf.constant([416, 416])):
    scores_max = tf.math.reduce_max(scores, axis=-1)
    scores_max = tf.math.reduce_max(scores_max, axis=0)  # New line
    mask = scores_max >= score_threshold
    mask = tf.repeat(tf.expand_dims(mask, 0), tf.shape(scores)[0], axis=0)  # New line
    class_boxes = tf.boolean_mask(box_xywh, mask)
    pred_conf = tf.boolean_mask(scores, mask)
    class_boxes = tf.reshape(class_boxes, [tf.shape(scores)[0], -1, tf.shape(class_boxes)[-1]])
    pred_conf = tf.reshape(pred_conf, [tf.shape(scores)[0], -1, tf.shape(pred_conf)[-1]])

    box_xy, box_wh = tf.split(class_boxes, (2, 2), axis=-1)

    input_shape = tf.cast(input_shape, dtype=tf.float32)

    box_yx = box_xy[..., ::-1]
    box_hw = box_wh[..., ::-1]

    box_mins = (box_yx - (box_hw / 2.)) / input_shape
    box_maxes = (box_yx + (box_hw / 2.)) / input_shape
    boxes = tf.concat([
        box_mins[..., 0:1],  # y_min
        box_mins[..., 1:2],  # x_min
        box_maxes[..., 0:1],  # y_max
        box_maxes[..., 1:2]  # x_max
    ], axis=-1)
    # return tf.concat([boxes, pred_conf], axis=-1)
    return (boxes, pred_conf)

[LC117]

Fixed it for me, thanks for sharing!

[chad-green]

Perfect, @jobpasin ! Thanks for posting your solution!

[aiswaryasukumar4]

@jobpasin Thanks for posting your solution. Fixed it for me!

[Amilitar]

It's Super! Thanks for the solution!!!!

[Anas-Alshaghouri]

Hello I am using this Yolov4 tensorflow to have live human detection with a USB camera. In case I want to use two or three cameras at the same time to do detections, they are working sequentially, in other words waiting each other. Does the solution of this issue help in this problem, although the function "filterbox" is not used in my code. Your help is highly appreciated.