Class changer visualization

[mstreit]

Class 1 and 2 will be defined by the user when selecting a cell in the confusion matrix (class 1=column, class 2=row).

Users should be able to select all bands in the parallel sets visualization, allowing them to specify union and intersections between classes. When selecting a band, the selected images should be retrieved from the server and displayed.

[NMO13]

Looks nice.

[mstreit]

d3 lib that can possibly by used: https://github.com/jasondavies/d3-parsets

[gfrogat]

facetted browsing idea, we could create dynamic sets/groups of classes. Would be a little bit more flexible as users could build their own "contrasts"

[NMO13]

According to our meeting notes:

Class changer images might be interesting as well This topic gets more and more important, because images representing the algorithm/solution the best can be used for publications Sepp hasn’t looked into that, but he finds it very interesting

[NMO13]

One question: When we want to show the track of instances over time, we need to store each instance and its classification for each epoch. This sounds like a huge amount of data. Is my assumption correct? Alternatively, we just store the necessary data for selected sample images.

[NMO13]

I was just told that we have the data available. So everything is alright.

[gfrogat]

We still need to come up with a design / layout for visualizing the class changers.

[gfrogat]

Also it should be general enough so we can track any type of data (not just images but also numeric vectors, graphs, etc...)

[thinkh]

User tasks and possible solutions

• Show images (or samples) for a selected class and a single epoch
  • Possible solution
  • Single run: simple image view
  • Multiple runs: images side-by-side
• Show images (or samples) for a selected class for an epoch range (over time)
  • Possible solution
  • Single run: parallel sets, as suggested above
  • Multiple runs: ?
• Track selected images for an epoch range
  • Requires interaction for image selection
  • Simple table with 2+ columns (epoch, run1, ...) and list the predicted class for each epoch -- differences could be highlighted
  • Limitations: Does not scale to a lot of selected images
  • Path could also be highlighted in the parallel sets
  • Maybe we can augment the confusion matrix with trajectories for the predicted classes
  • Limitations: Does not scale to a lot of selected images and will clutter easily

[NMO13]

Some notes regarding our concept:

1. When the user select a cell in the confusion matrix, the class changer vis should filter the images accordingly.
2. Initially, when no cell is selected, the Misclassification and Stability charts should be visible. The class changer vis should should not be visible because it is just showing a giant hairball.
3. We have to constrain the number of representative images below the class changer vis
4. We could sort/group the representative images by some criteria

[gfrogat]

[NMO13]

Each barchart represents the confusion of one class against all others. The vertical stacking of bar charts forms 1 epoch. For multiple epochs, the vertical stacking of bar charts is extended horizontally.

[NMO13]

• Show inverse of stability: How often does an image not change?
• Use ThemeRiver approach?
• Show class changers as vertical lines and arrows (similar to parallel coordinates but without cluttering)
• Use bar charts to encode the class changers (see image above)

[thinkh]

Alternative approach if we can use relative numbers: Ternary (or Triaxial) Plots

Image from http://ethanfosse.blogspot.co.at/2012/03/triaxial-graphs.html

[NMO13]

Next idea:

Each column encodes 1 epoch. The pairs of black/red line segments at the top and bottom encode the predicted class and the rest classes. The red/black/red line segment in the middle encodes the actual class. The line segments are separated by 1 pixel.

The black lines are instances that don't change from epoch e_x to epoch e_x+1 The red lines are either outgoing or incoming (this depends on the definition) instances. That means these are instances that will change classes from e_x to e_x+1.

Multiple Runs:

Interleave the bars:

The nice thing here is that the we can use the color coding of the timelines because each timeline represents a run. Diverging/converging patterns are easily observable with this approach.

Possible "special" effects:

• Hovering over a column leads to a fish-eye effect: Increase the line segments to bars and show representative images.
• or show representative images to the left of the plot
• show bands (lineup-like) between two selected columns and put representative images near the bands.

[thinkh]

Out of scope for the paper. Hence, moved to icebox.

[gfrogat]

Based on experiments with a Matplotlib Prototype this approach doesn't work.