precision very low, and recall is 1

[Edwardf0t1]

Hi Silvio,

I was able to run your Minute classifier program, and got reasonable performance in terms of mAP. I am also interested to see the precision and recall metric for each of "goal, card, and subs" event, so I enabled the precision and recall codes in your Network.py:

'

        with tf.name_scope("precision"):
            # self._precision, self._precision_update = tf.metrics.precision(labels=tf.argmax(self.labels,1), predictions=tf.argmax(self.logits,1), name='precision', )
            # self._reset_precision_op = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='precision'))

            self._precision_0, self._precision_update_0 = tf.metrics.precision(labels=self.labels_0, predictions=self.predictions_0, name='precision_0', )
            #self._reset_precision_op_0 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='precision_0'))

            self._precision_1, self._precision_update_1 = tf.metrics.precision(labels=self.labels_1, predictions=self.predictions_1, name='precision_1', )
            #self._reset_precision_op_1 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='precision_1'))

            self._precision_2, self._precision_update_2 = tf.metrics.precision(labels=self.labels_2, predictions=self.predictions_2, name='precision_2', )
            #self._reset_precision_op_2 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='precision_2'))

            self._precision_3, self._precision_update_3 = tf.metrics.precision(labels=self.labels_3, predictions=self.predictions_3, name='precision_3', )
            #self._reset_precision_op_3 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='precision_3'))

            self._batch_precision = tf.reduce_mean([self._precision_0, self._precision_1, self._precision_2, self._precision_3], name='batch_precision')
            self._precision = tf.Variable(0.0, trainable=False, name='precision')
            self._precision_update = tf.assign(self._precision,  tf.reduce_mean([self._precision_update_0, self._precision_update_1, self._precision_update_2, self._precision_update_3]), name='precision_update' )
            #self._reset_precision_op = tf.assign(self._precision, 0.0, name='reset_precision_op')

        # RECALL
        with tf.name_scope("recall"):
            self._recall_0, self._recall_update_0 = tf.metrics.recall(labels=self.labels_0, predictions=self.predictions_0, name='recall_0', )
            #self._reset_recall_op_0 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='recall_0'))

            self._recall_1, self._recall_update_1 = tf.metrics.recall(labels=self.labels_1, predictions=self.predictions_1, name='recall_1', )
            #self._reset_recall_op_1 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='recall_1'))

            self._recall_2, self._recall_update_2 = tf.metrics.recall(labels=self.labels_2, predictions=self.predictions_2, name='recall_2', )
            #self._reset_recall_op_2 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='recall_2'))

            self._recall_3, self._recall_update_3 = tf.metrics.recall(labels=self.labels_3, predictions=self.predictions_3, name='recall_3', )
            #self._reset_recall_op_3 = tf.variables_initializer(var_list=tf.get_collection(tf.GraphKeys.LOCAL_VARIABLES, scope='recall_3'))

            self._batch_recall = tf.reduce_mean([self._recall_0, self._recall_1, self._recall_2, self._recall_3], name='batch_recall')
            self._recall = tf.Variable(0.0, trainable=False, name='recall')
            self._recall_update = tf.assign(self._recall,  tf.reduce_mean([self._recall_update_0, self._recall_update_1, self._recall_update_2, self._recall_update_3]), name='recall_update' )`

What I got is very strange. On the test set: the precision is very low, and recall is 1 for each event: auc: 0.623 (auc_PR_0: 0.974 auc_PR_1: 0.532 auc_PR_2: 0.657 auc_PR_3: 0.680) precision: 0.251 (precision_0: 0.857 precision_1: 0.049 precision_2: 0.059 precision_3: 0.039) recall: 0.623 (recall_0: 1.000 recall_1: 1.000 recall_2: 1.000 recall_3: 1.000) Loss: 34.6 Accuracy: 0.645 mAP: 0.623

This is also the case for the train set and validation set.

Just wonder if I missed anything to use precision and recall metrics? Thanks.

[SilvioGiancola]

The AP is obtained as the AUC of the Recall/Precision curve. If you want a single Recall and Precision, you need to set a threshold to your predictions. My guess is that this threshold is set too low hence provides you with a perfect recall (you have spotted all instances) but your precision is very low (lot of false positive). I would not recommend using that piece of code for recall/precision, it was commented fora reason. The mAP metrics makes more sense for spotting.

Hope that help