Suggestion: Model Explainability Interpretability Visualization using Decision Tree Diagrams

[tauheedul]

I previously logged an enhancement for #511 - "Suggestion - Make Machine Learning Models explainable by design with ML.NET". @rogancarr kindly broke down that request into separate new enhancements. https://github.com/dotnet/machinelearning/issues/511#issuecomment-448329385

In this enhancement, I am suggesting we add visualization of Machine Learning decisions using a Decision Tree within Visual Studio. Now that ML.NET supports Feature Importance (#599) this should be possible.

How it could be done. I propose in the Visual Studio IDE Editor a new Model Visualization tab is added (Similar to the CPU Performance usage or a UML Class Diagram that shows how objects relate to each other)

• The tab could show a graphical decision tree showing a path the algorithm took and why it did it
• The decision tree could be derived from the feature importance metrics that are available within ML.NET

• At each level of a tree, you could inspect the Data that contributed to it (an extension of IDataView?)

  There have been similar issues logged, but my request is different from the following...

• Feature request: get reasons behind predictions made by Decision Trees #913
• Is it possible to visualize a generated decision tree? #326
• Pipeline visualization please #2478
• Proof of concept for debugger visualization #847

Advanced Deep Learning Debugging / Explainability Also from the perspective of a Deep Learning model e.g. a TensorFlow or ONNX model, would it be possible to peek into the neural network nodes? (not only a high-level feature importance score but peek into what each node is doing e.g. you can view feature importance at each node level)

In the Visual Studio IDE we could have a Model with a tree of neurons (maybe thousands)... each neuron can be expanded and the evaluation of what each node is doing can be inspected.

This can open up the opportunity to allow developers to create custom graphing for visual feedback and monitor the progression of a decision in real-time.

It allows you to trace which neurons are fired in any given execution of the model and how a given set of data impacted the score and at what stage it changed.

An example problem scenario is AI Healthcare diagnosis... Why a prediction or diagnosis was made? to avoid over-diagnosis and unnecessary costs

[rustd]

Thank you @tauheedul for this great write up. We will take this into our planning.

[tauheedul]

Thanks @rustd please feel free to tag me if you have any questions or would like to discuss an approach you're considering for this.

[TomFinley]

Hi @rustd, can you share more details? The closest thing that would allow us to do this as directly requested is this, but it seems possibly somewhat limited. Maybe there's something in the pipe I don't know about. (Actually I'm pretty sure there's plenty in the pipe I don't know about. 😉)

In the past what we've done is make things that could be displayed using WPF and HTML (in particular D3 based visualizations were my own personal favorite), but this was more appropriate when this code was backing a command line tool and GUI, and not a cross platform .NET API.