EnsembleMatrix: Interactive Visualization to Support Machine Learning with Multiple Classifier

[hon9g]

Abstract

• In this paper, we present EnsembleMatrix, an interactive visualization system
  • that presents a graphical view of confusion matrics
  • for exploring the space of combinations of classifiers.
  • to help users understand the relative merits of various classifiers.

Introduction

• As machine learning becomes more widely applied, both within HCI community and beyond,
• a critical human-computer interaction challenge is to provide adequate tools to allow non-experts to wield ML techniques effectively.

• In this paper, we restrict our attention to multiclass classification problems.

  PainPoint 1:

  • At present, a common applied machine learning workflow is to iteratively develop classifiers
• by refining low-level choices such as feature selection, algorithm selection, parameter tuning, and so on.
• This is usually performed using manual trial and error in an approach that represents hill-climbing through the model space.

PainPoint 2:

• Models are compared using accuracy, or other coarse summaries. These simple summaries provide little direction on how to improve the classifier.

• This problem exacerbated in multi-class problems where single value summaries, such as accuracy, can very misleading.

  Previous approach:

• To address this issue, focused on developing tools for better explaining a single classifier's behavior (or misbehavior).

Limitations:

• This workflow is often sub-optimal.
• It obscures the beneficial dependencies and complementarities exist between classifiers.

• and discard context of the space of possible models.

  • In the proposed workflow, we instead place emphasis on taking a holistic perspective provided by a set of models selected from the model space.

  • By supplying a visual summary that spans multiple classifiers, we help users understand various complimentary properties.

[hon9g]

• Details

  EnsembleMatrix

  a system that helps machine learning practitioners explore and combine individual classifiers.

Visualizing the Confusion Matrix:

• Confusion matrix as a graphical heat-map.
  • A confusion matrix represents classification results by plotting data instance in grid where the column is an instance's predicted class and row an instance's true class.
  • Confusion matrix can reveal trends and patterns in the classification results and by extension reveal behavior of classifier itself.
• Confusion matrix as the core visual element of our system.
  • since it can represent results from all classification algorithms and interpretation is relatively algorithm agnostic.
• Ordering of the matrix can greatly influence the patterns visible.
  • EnsembleMatrix orders each of the Component Classifier matrices independently to highlight sets of classes which are frequently confused by that particular classifier.
  • This corresponds to grouping clusters in an adjacency matrix.
  • Additionally, as users update the Ensemble Classifier view, the main matrix is reordered interactively.
  • This necessitates a fast reordering algorithm, so we chose to use the barycenter heuristic, borrowed from the layout of bipartite graphs.
• Small horizontal bars below each matrix in the Component Classifier view show the accuracy for each of these classifiers.
• A similar set of bars in the Linear Combination pane show the overall accuracy of the current Ensemble Classifier and the accuracy of the currently selected partition.

Interacting with EmsembleMatrix:

2 basic machanisms for users to explore combinations of the classifiers.

• Partitioning: Divides the class space into multiple partitions.

• Linear Combinations: Arbitarary linear combinations of the Component Classifiers for each of the partitions.

[hon9g]

User Study

[hon9g]

Personal Comment

• With EnsembleMatrix system, user still need to do iterative development ML model, but the it can help the iterative work less painful.