Mean average precision (mAP) metric

[0x00b1]

keras-rcnn should provide a mean average precision (mAP) Keras-compatiable metric that can be used to evaluate the performance of a model during training.

[jhung0]

def mAP(y_true, y_pred):
    return tensorflow.reduce_mean(tensorflow.metrics.sparse_average_precision_at_k(tensorflow.cast(y_true, tensorflow.int64), y_pred, 1)[0])

[0x00b1]

@jhung0 You’e really close. Two comments:

• Use keras.backend.mean rather than tensorflow.reduce_mean
• We need to implement a custom average precision function because tensorflow.metrics.sparse_average_precision_at_k expects to be updated in training (i.e. it’s incompatible with Keras’s per-batch metrics)

[0x00b1]

And send a PR! It’ll be easier to discuss. 😎

[0x00b1]

My simple NumPy implementation:

numpy.mean(numpy.cumsum(y_true[numpy.argsort(~y_pred)]) / numpy.cumsum(y_true[numpy.argsort(~y_true)]))

Does that look right?

[jhung0]

No, that wouldn't calculate it correctly. Here's a numpy implementation (but I think y_true and y_pred are not in one hot form) https://github.com/benhamner/Metrics/blob/master/Python/ml_metrics/average_precision.py

On Fri, Jun 2, 2017 at 11:31 AM, Allen Goodman notifications@github.com wrote:

My simple NumPy implementation:

numpy.mean(numpy.cumsum(y_true[numpy.argsort(~y_pred)]) / numpy.cumsum(y_true[numpy.argsort(~y_true)]))

Does that look right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/broadinstitute/keras-rcnn/issues/6#issuecomment-305822070, or mute the thread https://github.com/notifications/unsubscribe-auth/AJJbgmfc8BMqeroiROVqQ-ST1dgL_SQpks5sACrCgaJpZM4NqVAL .

[jhung0]

my numpy implementation (for y in one hot form)

 _, classes = y_true.shape

average_precisions = []

for index in range(classes):
        row_indices_sorted = numpy.argsort(-y_pred[:, index])

        y_true_cls = y_true[row_indices_sorted, index]
        y_pred_cls = y_pred[row_indices_sorted, index]

        tp = (y_true_cls == 1)
        fp = (y_true_cls == 0)

        fp = numpy.cumsum(fp)
        tp = numpy.cumsum(tp)

        npos = numpy.sum(y_true_cls)

        rec = tp*1.0 / npos

        # avoid divide by zero in case the first detection matches a difficult
        # ground truth
        prec = tp*1.0 / numpy.maximum((tp + fp), numpy.finfo(numpy.float64).eps)

        mrec = numpy.concatenate(([0.], rec, [1.]))
        mpre = numpy.concatenate(([0.], prec, [0.]))

        # compute the precision envelope
        for i in range(mpre.size - 1, 0, -1):
            mpre[i - 1] = numpy.maximum(mpre[i - 1], mpre[i])

        # to calculate area under PR curve, look for points
        # where X axis (recall) changes value
        i = numpy.where(mrec[1:] != mrec[:-1])[0]

        # and sum (\Delta recall) * prec
        average_precisions.append(numpy.sum((mrec[i + 1] - mrec[i]) * mpre[i + 1]))

[jiahuei]

It can be done by just using TF's own function, tf.metrics.sparse_average_precision_at_k.

Assuming y_true is a tensor of shape (batch_size, num_labels), and y_pred is of shape (batch_size, num_classes):

_, m_ap = tf.metrics.sparse_average_precision_at_k(y_true, y_pred, k)

For example:

y_true = np.array([[2], [1], [0], [3], [0], [1]]).astype(np.int64)
y_true = tf.identity(y_true)

y_pred = np.array([[0.1, 0.2, 0.6, 0.1],
                   [0.8, 0.05, 0.1, 0.05],
                   [0.3, 0.4, 0.1, 0.2],
                   [0.6, 0.25, 0.1, 0.05],
                   [0.1, 0.2, 0.6, 0.1],
                   [0.9, 0.0, 0.03, 0.07]]).astype(np.float32)
y_pred = tf.identity(y_pred)

_, m_ap = tf.metrics.sparse_average_precision_at_k(y_true, y_pred, 3)

sess = tf.Session()
sess.run(tf.local_variables_initializer())

tf_map = sess.run(m_ap)

tf_map will have the correct value of 0.3611.

[b18arundhati]

Hi!

I have tried out the following keras implementation for the mAP metric (in a multilabel classification dataset). Could you kindly take a look if this seems all right?

def calculate_mAP(y_true,y_pred):
    num_classes = y_true.shape[1]
    average_precisions = []
    relevant = K.sum(K.round(K.clip(y_true, 0, 1)))
    tp_whole = K.round(K.clip(y_true * y_pred, 0, 1))
    for index in range(num_classes):
        temp = K.sum(tp_whole[:,:index+1],axis=1)
        average_precisions.append(temp * (1/(index + 1)))
    AP = Add()(average_precisions) / relevant
    mAP = K.mean(AP,axis=0)
    return mAP

[mrStasSmirnoff]

@jhung0 it looks exactly like Matlab implementation of VOCdevkit! Could you please give some comments to your code, especially to the first half?

[lanhongvp]

I try to give some specific explanations about @b18arundhati code to help others understand this example. The tf.metrics.sparse_average_precision_at_k will be replaced by tf.metrics.average_precision_at_k. And by browsing the code in tensorflow, you will find that when your inputs are y_true and y_pred, this function will actually transform the y_pred to y_pred_idx, by using top_k function.

y_true is a tensor of shape (batch_size, num_labels), and y_pred is of shape (batch_size, num_classes)

import tensorflow as tf
import numpy as np

y_true = np.array([[2], [1], [0], [3], [0]]).astype(np.int64)
y_true = tf.identity(y_true)

y_pred = np.array([[0.1, 0.2, 0.6, 0.1],
                   [0.8, 0.05, 0.1, 0.05],
                   [0.3, 0.4, 0.1, 0.2],
                   [0.6, 0.25, 0.1, 0.05],
                   [0.1, 0.2, 0.6, 0.1]
                   ]).astype(np.float32)
y_pred = tf.identity(y_pred)

_, m_ap = tf.metrics.sparse_average_precision_at_k(y_true, y_pred, 3)

sess = tf.Session()
sess.run(tf.local_variables_initializer())

stream_vars = [i for i in tf.local_variables()]

tf_map = sess.run(m_ap)
print(tf_map)

print((sess.run(stream_vars)))

tmp_rank = tf.nn.top_k(y_pred,3)

print(sess.run(tmp_rank)) 

I add stream_vars = [i for i in tf.local_variables()] so that you can see the two local_variables which is created in this tf.metrics.sparse_average_precision_at_k function.

And I add tmp_rank = tf.nn.top_k(y_pred,3) in order to help you understand by changing the value of k ,the prediction index which is used in tf.metrics.sparse_average_precision_at_k . You can change the value of k to see the different result, and the tmp_rank represents the index which is used in calculating the average precision.

For example, when k=1, only the first batch match the label, so the average precision at 1 result will be 1/6 = 0.16666666. When k=2, the third batch will also match the label, so the average precision at 2 result will be (1+(1/2))/6=0.25.

[freygit]

I try to give some specific explanations about @b18arundhati code to help others understand this example. The tf.metrics.sparse_average_precision_at_k will be replaced by tf.metrics.average_precision_at_k. And by browsing the code in tensorflow, you will find that when your inputs are y_true and y_pred, this function will actually transform the y_pred to y_pred_idx, by using top_k function.

y_true is a tensor of shape (batch_size, num_labels), and y_pred is of shape (batch_size, num_classes)

import tensorflow as tf
import numpy as np

y_true = np.array([[2], [1], [0], [3], [0]]).astype(np.int64)
y_true = tf.identity(y_true)

y_pred = np.array([[0.1, 0.2, 0.6, 0.1],
                   [0.8, 0.05, 0.1, 0.05],
                   [0.3, 0.4, 0.1, 0.2],
                   [0.6, 0.25, 0.1, 0.05],
                   [0.1, 0.2, 0.6, 0.1]
                   ]).astype(np.float32)
y_pred = tf.identity(y_pred)

_, m_ap = tf.metrics.sparse_average_precision_at_k(y_true, y_pred, 3)

sess = tf.Session()
sess.run(tf.local_variables_initializer())

stream_vars = [i for i in tf.local_variables()]

tf_map = sess.run(m_ap)
print(tf_map)

print((sess.run(stream_vars)))

tmp_rank = tf.nn.top_k(y_pred,3)

print(sess.run(tmp_rank)) 

I add stream_vars = [i for i in tf.local_variables()] so that you can see the two local_variables which is created in this tf.metrics.sparse_average_precision_at_k function.

And I add tmp_rank = tf.nn.top_k(y_pred,3) in order to help you understand by changing the value of k ,the prediction index which is used in tf.metrics.sparse_average_precision_at_k . You can change the value of k to see the different result, and the tmp_rank represents the index which is used in calculating the average precision.

For example, when k=1, only the first batch match the label, so the average precision at 1 result will be 1/6 = 0.16666666. When k=2, the third batch will also match the label, so the average precision at 2 result will be (1+(1/2))/6=0.25.

your explanation should be : For example, when k=1, only the first batch match the label, so the average precision at 1 result will be 1/5 = 0.2. NOT: 1/6 = 0.16666666 When k=2, the third batch will also match the label, so the average precision at 2 result will be (1+(1/2))/5=0.3. NOT: (1+(1/2))/6=0.25

[sushantMoon]

As an addition to the discussion, I recently found out this library called extra-keras-metrics which provides average_precision_at_k along with the rest of the metrics from Python APIs of tf.metrics.