[feature request] confusion matrix visualization

[anne1994]

I was trying to display a confusion matrix on tensorboard for four classes and i get the output as below.

can i print the values of them ? also the labels ? i added tf.confusion_matirx(predictions,labels) in the eval_image_classifier.py (for slim/inceptionv3) . Also can i print the complete output on command line? I am able to show something like this but i want to see the entire array/matrix. any ideas?

[chihuahua]

TensorBoard has a text dashboard that supports HTML strings. That means you can pass strings to tf.summary.text and have them show up in the text dashboard.

@dandelionmane, where is the text dashboard documented?

[anne1994]

@chihuahua does it only work for tensorflow 1.0 ? I have the older version tensorflow 0.12.1 installed on my gpu. It says op=tf.summary.text("confusion_matrix",c_matrix) AttributeError: 'module' object has no attribute 'text'

[chihuahua]

@anne1994, installing TensorFlow 1.3 (specifically, pip installing one of these wheels) should solve that problem.

[hkhatod]

we need this in tensorflow https://rare-technologies.com/wp-content/uploads/2017/10/facets_dive_20newsgroups-widget_only.html

[hkhatod]

Here is something I have put together which give visual and well as normalized numeric cm.

[jmoraleda]

What is the currently recommended way of computing a confusion matrix during evaluation mode when using estimators so that it can later be viewed in tensorboard, for an experience similar to the image shown above?

I have considered constructing a custom metric leveraging the function _streaming_confusion_matrix in https://github.com/tensorflow/tensorflow/blob/r1.8/tensorflow/python/ops/metrics_impl.py. But it is not immediately apparent to me how to then have the estimator save a SummaryImage based on the final confusion matrix tensor. In addition, this approach uses a private function, which hardly seems best practices, and writing custom metrics also is discouraged, so I wonder what is the recommended approach.

[nfelt]

@jmoraleda I believe you can directly output a tf.Summary protocol buffer as a metric output from Estimator, and it should save it as a summary that can be visualized in TensorBoard. E.g. if you use tf.summary.image() you can emit the output as a metric.

I can't speak to whether TensorFlow might be open to exposing _streaming_confusion_matrix in some form or what the status is for custom metrics - you might have to ask on their StackOverflow or GitHub about what they recommend.

[jmoraleda]

Thank you @nfelt. I found what I think is a reasonable elegant solution that requires no modifications to tensorboard. I tested it in tensorflow 1.8 using python3.

The key insight for me was two fold, one that there is no need to implement any custom metric because one can use the confusion matrix operation that is computed as part of the included tf.metrics.mean_iou metric and second that this operation can be accessed by a custom SessionRunHook to save a tf.Summary.Image.

I append a full implementation of all this. The code of my custom SessionRunHook that saves the tf.Summary is directly based on the code @hkhatod references above.

Create the the custom hook object to be passed to the EvalSpec of the estimator:

confusionMatrixSaveHook = my_package.confusion_matrix.SaverHook(
      labels = ['First class','Second class','Third class'],
      confusion_matrix_tensor_name = 'mean_iou/total_confusion_matrix',
      summary_writer = tf.summary.FileWriterCache.get(str(MODEL_DIR / "eval"))
)

The module my_package.confusion_matrix is:

import tensorflow as tf
import numpy as np

import textwrap
import re
import io
import itertools
import matplotlib

class SaverHook(tf.train.SessionRunHook):
    """
    Saves a confusion matrix as a Summary so that it can be shown in tensorboard
    """

    def __init__(self, labels, confusion_matrix_tensor_name, summary_writer):
        """Initializes a `SaveConfusionMatrixHook`.

        :param labels: Iterable of String containing the labels to print for each
                       row/column in the confusion matrix.
        :param confusion_matrix_tensor_name: The name of the tensor containing the confusion
                                             matrix
        :param summary_writer: The summary writer that will save the summary
        """
        self.confusion_matrix_tensor_name = confusion_matrix_tensor_name
        self.labels = labels
        self._summary_writer = summary_writer

    def end(self, session):
        cm = tf.get_default_graph().get_tensor_by_name(
                self.confusion_matrix_tensor_name + ':0').eval(session=session).astype(int)
        globalStep = tf.train.get_global_step().eval(session=session)
        figure = self._plot_confusion_matrix(cm)
        summary = self._figure_to_summary(figure)
        self._summary_writer.add_summary(summary, globalStep)

    def _figure_to_summary(self, fig):
        """
        Converts a matplotlib figure ``fig`` into a TensorFlow Summary object
        that can be directly fed into ``Summary.FileWriter``.
        :param fig: A ``matplotlib.figure.Figure`` object.
        :return: A TensorFlow ``Summary`` protobuf object containing the plot image
                 as a image summary.
        """

        # attach a new canvas if not exists
        if fig.canvas is None:
            matplotlib.backends.backend_agg.FigureCanvasAgg(fig)

        fig.canvas.draw()
        w, h = fig.canvas.get_width_height()

        # get PNG data from the figure
        png_buffer = io.BytesIO()
        fig.canvas.print_png(png_buffer)
        png_encoded = png_buffer.getvalue()
        png_buffer.close()

        summary_image = tf.Summary.Image(height=h, width=w, colorspace=4,  # RGB-A
                                      encoded_image_string=png_encoded)
        summary = tf.Summary(value=[tf.Summary.Value(tag=self.confusion_matrix_tensor_name, image=summary_image)])
        return summary

    def _plot_confusion_matrix(self, cm):
        '''
        :param cm: A confusion matrix: A square ```numpy array``` of the same size as self.labels
    `   :return:  A ``matplotlib.figure.Figure`` object with a numerical and graphical representation of the cm array
        '''
        numClasses = len(self.labels)

        fig = matplotlib.figure.Figure(figsize=(numClasses, numClasses), dpi=100, facecolor='w', edgecolor='k')
        ax = fig.add_subplot(1, 1, 1)
        ax.imshow(cm, cmap='Oranges')

        classes = [re.sub(r'([a-z](?=[A-Z])|[A-Z](?=[A-Z][a-z]))', r'\1 ', x) for x in self.labels]
        classes = ['\n'.join(textwrap.wrap(l, 20)) for l in classes]

        tick_marks = np.arange(len(classes))

        ax.set_xlabel('Predicted')
        ax.set_xticks(tick_marks)
        ax.set_xticklabels(classes, rotation=-90, ha='center')
        ax.xaxis.set_label_position('bottom')
        ax.xaxis.tick_bottom()

        ax.set_ylabel('True Label')
        ax.set_yticks(tick_marks)
        ax.set_yticklabels(classes, va='center')
        ax.yaxis.set_label_position('left')
        ax.yaxis.tick_left()

        for i, j in itertools.product(range(numClasses), range(numClasses)):
            ax.text(j, i, int(cm[i, j]) if cm[i, j] != 0 else '.', horizontalalignment="center", verticalalignment='center', color="black")
        fig.set_tight_layout(True)
        return fig

If one's estimator is not computing the mean_iou metric already, which will be the most common case, it can be added as:

estimator = tf.contrib.estimator.add_metrics(estimator, 
      lambda labels, predictions: {'mean_iou': tf.metrics.mean_iou(labels, predictions['class_ids'],3)})

(In this example there are three possible outcomes to the classifier, hence the 3 at the end)

[nfelt]

@jmoraleda Glad you figured out something that works!

[hkhatod]

@jmoraleda This is so much more elegant. Thanks for improving my code. Thanks!!!

[ldalzovo]

I am trying to use the code provided by @jmoraleda in Tensorflow Object Detection API, but I have an error executing estimator = tf.contrib.estimator.add_metrics(estimator, lambda labels, predictions: {'mean_iou': tf.metrics.mean_iou(labels, predictions['class_ids'],3)}) with this message: lambda labels, predictions: {'mean_iou': tf.metrics.mean_iou(labels, predictions['class_ids'],60)}) KeyError: 'class_ids'

Any idea how to solve this or suggestion how to debug it? Thank you!

[Ouwen]

@ldalzovo you probably want to check what the predictions tensor is. It looks like the ['class_ids'] key on predictions does not exist. You should make sure the inputs to the function are following the tensorflow docs for tf.metrics.mean_iou

labels: A Tensor of ground truth labels with shape [batch size] and of type int32 or int64. The tensor will be flattened if its rank > 1. predictions: A Tensor of prediction results for semantic labels, whose shape is [batch size] and type int32 or int64. The tensor will be flattened if its rank > 1.

@jmoraleda and @hkhatod great work on the solutions provided for visualizing confusion matrices.

I wish this was natively supported

[kbegiedza]

Thanks @jmoraleda for nice solution.

How can I visualize conf matrix with research/object_detection/model_main_tf2.py ?

[Annieliaquat]

Thanks @jmoraleda for nice solution.

How can I visualize conf matrix with research/object_detection/model_main_tf2.py ?

Did you find any solution for your this problem.? Because I also want to find the confusion matrix for ssd-mobnet from Tensorflow object detection API?