COCO METRICS broken

[hugopi]

Hi kerasCV,

I am using keras cv since v0.4 and i have worked a lot with coco metrics.

I have updated my keras CV version from v0.5 to pip install git+https://github.com/keras-team/keras-cv.git and know I have an error with coco metrics.

I used it like in your tutorial :

class EvaluateCOCOMetricsCallback(tf.keras.callbacks.Callback):
    def __init__(self, data, logs_dir):
        super().__init__()
        self.data = data
        self.metrics = keras_cv.metrics.BoxCOCOMetrics(
            bounding_box_format="xywh",
            # passing 1e9 ensures we never evaluate until
            # `metrics.result(force=True)` is
            # called.
            evaluate_freq=1e9,
        )
        self.logs_dir = logs_dir

    def on_epoch_end(self, epoch, logs):
        self.metrics.reset_state()
        for batch in tqdm.tqdm(self.data):
            images, y_true = batch[0], batch[1]
            y_pred = self.model.predict(images, verbose=0)
            self.metrics.update_state(y_true, y_pred)

        result = self.metrics.result(force=True)

        with tf.summary.create_file_writer(self.logs_dir).as_default():
            for metric in result.items():
                metric_name, metric_value = metric
                tf.summary.scalar(metric_name, metric_value, step=epoch)

        logs.update(result)
        return logs

And in call backs : coco_metrics = EvaluateCOCOMetricsCallback(self.val_ds.take(10), logs_dir)

It was working perfectly before updating and now i have this error :

  2023-07-31 14:23:41.535556: W tensorflow/core/framework/op_kernel.cc:1816] UNKNOWN: InvalidArgumentError: {{function_node __wrapped__ConcatV2_N_10_device_/job:localhost/replica:0/task:0/device:CPU:0}} ConcatOp : Dimension 1 in both shapes must be equal: shape[0] = [32,5,4] vs. shape[1] = [32,3,4] [Op:ConcatV2] name: concat
Traceback (most recent call last):

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 266, in __call__
    return func(device, token, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 144, in __call__
    outputs = self._call(device, args)
              ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 151, in _call
    ret = self._func(*args)
          ^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 643, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 201, in result_on_host_cpu
    return tf.constant(obj_result(force), obj.dtype)
                       ^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 252, in result
    self._cached_result = self._compute_result()
                          ^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 260, in _compute_result
    _box_concat(self.ground_truths),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 43, in _box_concat
    result[key] = tf.concat([b[key] for b in boxes], axis=0)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 6656, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__ConcatV2_N_10_device_/job:localhost/replica:0/task:0/device:CPU:0}} ConcatOp : Dimension 1 in both shapes must be equal: shape[0] = [32,5,4] vs. shape[1] = [32,3,4] [Op:ConcatV2] name: concat

Traceback (most recent call last):
  File "/snap/pycharm-community/342/plugins/python-ce/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 1, in <module>
  File "/snap/pycharm-community/342/plugins/python-ce/helpers/pydev/_pydev_bundle/pydev_umd.py", line 198, in runfile
    pydev_imports.execfile(filename, global_vars, local_vars)  # execute the script
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/snap/pycharm-community/342/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/home/hugo/devWorkingPlace/src/rd-ship-detection/Model_Factory/training_pipeline.py", line 254, in <module>
    training_pipline()
  File "/home/hugo/devWorkingPlace/src/rd-ship-detection/Model_Factory/training_pipeline.py", line 44, in __call__
    training_history = self.training()
                       ^^^^^^^^^^^^^^^
  File "/home/hugo/devWorkingPlace/src/rd-ship-detection/Model_Factory/training_pipeline.py", line 222, in training
    history = self.model.fit(
              ^^^^^^^^^^^^^^^
  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras/src/utils/traceback_utils.py", line 70, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/hugo/devWorkingPlace/src/rd-ship-detection/Model_Factory/pipelines_utils/coco_metrics.py", line 26, in on_epoch_end
    result = self.metrics.result(force=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 206, in result_fn
    py_func_result = tf.py_function(
                     ^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnknownError: {{function_node __wrapped__EagerPyFunc_Tin_1_Tout_1_device_/job:localhost/replica:0/task:0/device:CPU:0}} InvalidArgumentError: {{function_node __wrapped__ConcatV2_N_10_device_/job:localhost/replica:0/task:0/device:CPU:0}} ConcatOp : Dimension 1 in both shapes must be equal: shape[0] = [32,5,4] vs. shape[1] = [32,3,4] [Op:ConcatV2] name: concat
Traceback (most recent call last):

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 266, in __call__
    return func(device, token, args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 144, in __call__
    outputs = self._call(device, args)
              ^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/ops/script_ops.py", line 151, in _call
    ret = self._func(*args)
          ^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/autograph/impl/api.py", line 643, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 201, in result_on_host_cpu
    return tf.constant(obj_result(force), obj.dtype)
                       ^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 252, in result
    self._cached_result = self._compute_result()
                          ^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 260, in _compute_result
    _box_concat(self.ground_truths),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/keras_cv/metrics/object_detection/box_coco_metrics.py", line 43, in _box_concat
    result[key] = tf.concat([b[key] for b in boxes], axis=0)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None

  File "/home/hugo/miniconda3/envs/sinbad_build/lib/python3.11/site-packages/tensorflow/python/framework/ops.py", line 6656, in raise_from_not_ok_status
    raise core._status_to_exception(e) from None  # pylint: disable=protected-access
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

tensorflow.python.framework.errors_impl.InvalidArgumentError: {{function_node __wrapped__ConcatV2_N_10_device_/job:localhost/replica:0/task:0/device:CPU:0}} ConcatOp : Dimension 1 in both shapes must be equal: shape[0] = [32,5,4] vs. shape[1] = [32,3,4] [Op:ConcatV2] name: concat

 [Op:EagerPyFunc] name: 

I have also update my tensorflow from 2.11 to 2.13

Do you have change something on coco metrics ? Or going to tf 2.13 and his new tf.function features could broke coco metrics?
Thanks a lot for your fantastic work !

Hugo PIRES

[jbischof]

@hugopi you can please attach a minimal colab replicating the issue?

[robertorovella91]

I have the same problem with tensorlfow 2.12.0 and keras_cv 0.6.1 with yolov8 model

[CooperColdwell]

I had the same problem following the YOLOv8 tutorial with my own data and TF 2.10.0, KerasCV 0.5.1 (latest on conda-forge), although mine happens in result = self.metrics.result(force=True)

[ianstenbit]

We have an update coming soon to the OD tutorial which will get rid of this EvaluateCOCOMetricsCallback. In the meantime, please use the keras_cv.callbacks.PyCOCOCallback instead (this is also what will be used in the updated guide)

[silvada95]

We have an update coming soon to the OD tutorial which will get rid of this EvaluateCOCOMetricsCallback. In the meantime, please use the keras_cv.callbacks.PyCOCOCallback instead (this is also what will be used in the updated guide)

Any expected date for when this tutorial will be ready? I am having problems using both BoxCocoMetrics and PyCOCOCallback. None of the examples are working, from the KERAS website to the ones in this repo. When bugs don't happen the mAP keeps really low.

[ianstenbit]

Updated guide is available here: https://github.com/keras-team/keras-io/pull/1487 (not yet finalized)

[luisliborio]

Hey guys, do you have any udpates on this issue?

I tried to reproduce this tutorial Keras-yolov8 detection, and got exact the same problem using keras_cv.metrics.BoxCOCOMetrics() and as

I had the same problem following the YOLOv8 tutorial with my own data and TF 2.10.0, KerasCV 0.5.1 (latest on conda-forge), although mine happens in result = self.metrics.result(force=True)

I notice the same behaviour, it breaks when .result(force=True).

---

tf version: 2.14.0 keras_cv version: 0.7.1

[kvlsky]

Hey guys, do you have any updates on this issue?