Confusion Matrix for multi-class classifiers

[NealHumphrey]

Create a ScoreVisualizer that creates a heatmap color coded confusion matrix showing true label vs. predicted label for multi-class classifiers. Goal of heatmap is to better identify frequently-confused categories.

• Raw number version (heatmap body shows counts)
• percent-of-true version (i.e. percent of row or percent of column, depending on orientation of predicted vs. true)
• could implement smart color coding, i.e. more than just a single-color heatmap where 100% accurate and other relevant threshold levels get different color treatment.
• could implement precision / recall scores at the ends of columns/rows, similar to a joint plot. Maybe use a small pie chart of the score (since heatmap is already used for the number of items in each block).

[NealHumphrey]

@rebeccabilbro another option for our Visualizer this sprint.

[rebeccabilbro]

@NealHumphrey this sounds a bit like the one we have here. Would this be a subclass of ClassificationReport?

Re: for our Visualizer for this sprint, I'd like us to try to stick to user-driven design. We want to look for blog posts, articles, etc where machine learning practitioners are using visualizations to guide their model selection/evaluation process. Then we can think about how to implement those visualizations in the Yellowbrick API. The advantage of doing it this way is that we can address a specific use case, which is helpful for scoping purposes. Whatever preliminary implementation we build can be generalized in later sprints.

Let me know if you find any interesting posts or articles that you think might work!

[bbengfort]

Just wanted to add a couple of things to this issue, but don't want to get in the way of your conversation regarding this Sprint (so feel free to ignore the following):

• sklearn metrics confusion matrix heatmap
• multi-class confusion matrix for text mining

Question: how do we add k-fold cross-validation or something like cross_val_predict to the mix?

[rebeccabilbro]

A few related posts I found:

• http://notmatthancock.github.io/2015/10/28/confusion-matrix.html
• https://www.wolfram.com/mathematica/new-in-10/highly-automated-machine-learning/visualize-the-confusion-matrix-of-a-classifier.html
• http://oobaloo.co.uk/visualising-classifier-results-with-ggplot2
• https://research.tableau.com/sites/default/files/chi2009-talbot-ensemblematrix.pdf
• http://stats.stackexchange.com/questions/221863/how-to-visualize-classifier-output

@NealHumphrey thoughts?

[NealHumphrey]

The EnsembleMatrix is an interesting extension of this, and a great example of the visual steering. Should definitely dig into that more.

To start with it probably makes sense to just do this as the basic version, similar to the sklearn example Ben linked. What about:

• Default version is for one train/test split.
• Default is for normalized view (i.e. %), but option is for raw numbers in each class
• Option for running a kfold version of the visualization (this must output normalized view)

Down the line, consider something like the ensemble version, realizing that is for a more niche application.

@rebeccabilbro re: your comment on basing this on existing articles/blog etc for this sprint, I did suggest this based on finding it in existing published approaches. Without an internet connection while traveling I couldn't search for blogs, so I pulled this from the Python Data Science Handbook - you can see the example from the book at the end of this Jupyter Notebook. With the user study example I picked I also had a multi-class classification and this visualization ended up being one that I wanted to have during my analysis, as well.

Not sure if this makes sense as a subclass of ClassificationReport? I'd need to dig into the code a bit.

[rebeccabilbro]

@NealHumphrey gotcha - thanks for sending the example from the Python Data Science Handbook. I was having trouble picturing what you had in mind! It sounds like you'd like us to reimplement the Seaborn heatmap as a Yellowbrick ConfusionMatrixVisualizer so that it can be used within the Yellowbrick API for Scikit-Learn confusion matrices?

Note: I think seeing your user study will help me to understand the specific requirements for this a bit better. Can you point me to which multi-class dataset you're working with, and explain the problems you encountered with interpreting the Sklearn confusion matrix? Is your notebook at a point where you can do a pull request?

The next step will be to sketch out how this will work with Yellowbrick's API. When you have a chance (and a steady internet connection), take a look at

• the source code for the Seaborn _HeatMapper
• the source code for Scikit-Learn's confusion_matrix
• the Yellowbrick ClassificationScoreVisualizer class, which the ConfusionMatrixVisualizer will inherit from

Then let's chat!

[rebeccabilbro]

Pulling into ready; sounds like this is the card @NealHumphrey and I will be working on for #125

[rebeccabilbro]

Another perspective to consider:

https://bitbucket.org/fraka6/mlboost/src/f4a54e980f24074a34697e386b9ef747247843cc/mlboost/util/confusion_matrix.py?at=default&fileviewer=file-view-default

[bbengfort]

See #144

[bbengfort]

See #144

[NealHumphrey]

Closed in #144

[fraka6]

https://bitbucket.org/fraka6/mlboost/src/f4a54e980f24074a34697e386b9ef747247843cc/mlboost/util/confusion_matrix.py?at=default&fileviewer=file-view-default

Here is the info about the packaging I have done to make confusion matrix easy to generate: https://fraka6.blogspot.com/2013/05/generating-confusion-matrix-great.html