Week 8. May. 10: Multi-Modal Learning - Orienting

[JunsolKim]

“Integrative Learning - Multi-modal, Complex and Complete Models” & “Epilogue: You in the Loop” in Thinking with Deep Learning, Chapters 17 & Epilogue; “Explainable Deep Learning: Concepts, Methods, and New Developments” by Wojciech Samek in Explainable Deep Learning (2023).

[kceeyang]

In the chapter “Explainable Deep Learning,” the author introduced different explanation methods and demonstrated the role of ‘explainability’ in the field of AI, as it helps people understand the behavior and underlying decision-making process of complex neural networks. Although the author listed several desiderata for explanation, the evaluation of explainable AI remains challenging. I’m curious, what is the biggest difficulty of designing a validation framework or a benchmark to measure the explanation quality?

[erikaz1]

Seeing that both post-hoc and ante-hoc approaches in XAI for making DL models more interpretable have been around for some time and are still actively researched areas, are public concerns related to “black box” AI better interpreted as a lack of understanding or misrepresentation of the (largely) post-hoc explanation process? The categories of attribution methods described by Samek (2023) in Explainable Deep Learning Chapter 2 are novel yet make sense to me. And as Samek states, “misunderstandings can occur…if we expect the XAI method to generate human-like or formal descriptions of model behavior” (9).

[guanhongliu2000]

I am curious about the evaluation of xxplanation quality: The article discusses various desiderata for explanations such as faithfulness and continuity. What are the best practices and metrics for evaluating these properties in deep learning models, especially in high-stakes domains like medical diagnostics or autonomous driving?

[XueweiLi1027]

Considering the discussion on the limitations of current XAI methods and the potential for future development, what are the most pressing challenges in making XAI methods more effective for non-expert stakeholders, and how might the field of AI address these challenges to improve the trustworthiness and usability of AI systems in high-stakes applications?

[HongzhangXie]

The Explainable Deep Learning: Concepts, Methods, and New Developments discusses four main methods of XAI (Explainable Artificial Intelligence) as follows:

Perturbation-based methods operate by altering parts of the input data, such as pixels or words, and then observing changes in the model output to understand the model's behavior. The core idea of this method is that if the output of the model changes significantly after perturbing a feature, that feature is crucial for the model's decision-making process.

Gradient-based methods use the gradient information of the model to determine the importance of input features. For instance, Grad-CAM utilizes the relationship between the gradients and feature maps in the final layers to explain the basis of the model's visual decisions.

Surrogate-based methods involve training a simple, interpretable model, such as a decision tree or linear model, to approximate the behavior of a complex model. By interpreting this simpler model, we can indirectly understand the decision-making process of the original complex model.

Propagation-based methods provide explanations by analyzing the flow of information within the model. Operations between layers (e.g., ReLU applied to a weighted sum of inputs) are easier to explain than the highly nonlinear prediction function represented by the entire model.

The author uses image data as an example. However, my question might not be well-suited for image data. I am considering a scenario with fewer variables, such as predicting a company's likelihood of bankruptcy using its financial data. Could we create a discrete grid of all variable combinations and use a deep model to predict outcomes for different variable combinations to explain the deep learning model? This approach is somewhat similar to perturbation-based methods. I believe it could avoid out-of-manifold effects and restrictive locality assumptions, while the granularity of the grid could be adjusted to manage computational demands.

[Xtzj2333]

Would it be possible to apply XAI in the 'Explainable Deep learning' chapter to interpret neural networks built for survey data? Perhaps we could utilize Perturbation-based Methods or Layer-wise Relevance Propagation?

[beilrz]

I think this is a very interesting paper to read about XAI. My questions, how is it possible to use XAI technique to explain association or interaction of features. This is also an issue author mentioned in the limitation section, but I believe this is very significant issue for modern architectures (for example, transformer attend to different parts of the input, based on the content of the input). Currently, the methods in this papers and the proposed future methods can only handle individual features, or features with few interaction, which are unlikely sufficient complex architectures , and the finding is hardly generalizable.

[CYL24]

In the chapter "Explainable Deep Learning," the author mentions that constructing explanations that are useful for human operators is crucial for enhancing human-AI interaction. Then in the context of human-AI interaction, what are some key considerations for designing interactive explanations that are both informative and user-friendly? Are there any challenges associated with ensuring that users can effectively utilize these explanations in practice?

[HamsterradYC]

The paper talks about how after converting a non-neural network model into a neural network model, the explanation is still based on the converted neural network rather than the original model. Could this conversion potentially introduce misleading explanations that do not accurately reflect the decision logic of the original model?

[Marugannwg]

This is the question I continuously have about deep learning (and explaining deep learning to other people who doubt it): "Even if we have full knowledge/interpretability input data and intuitive output, the process still feels like a "black box" where the internal operations are not directly interpretable... How to persuade your colleagues or your stakeholders?"

After reading the chapter, I kind feel we have considers this question along with our process of creating our model. Different audiences might need different types of explanations depending on their technical expertise and the decision's impact. Sounds like it is doomed if we have a hard-to-unexplainable model, and we need to embed explainability into the model training processes...

e.g.

• we probably needs aggregate simpler and conceptually straightforward models (Surrogate Models) for a student to learn;
• when presenting to stalkholders, models with clear visualization capacity or simple feature attribution might work better.

Is this some issue frequently considered in the lab/industry settings?

[maddiehealy]

In Explainable Deep Learning, I am interested in how researchers gauge what the threshold is for human understandability?

[La5zY]

In the context of explainable AI (XAI), the chapter discusses various methods for making deep learning models more interpretable, which method or combination of methods would provide the most practical balance between interpretability and maintaining model performance, and why?

[Pei0504]

The chapter on "Explainable Deep Learning" delves into the realm of explainable AI (XAI), emphasizing the necessity of making machine learning models transparent, as they are often perceived as black boxes. It explores a variety of post hoc explanation methods, discusses the integration of these explanations into model training to enhance performance and trustworthiness, and highlights the latest advancements in XAI. Particularly, it focuses on methods for transforming complex models to improve generalization and efficiency and introduces the concept of "neuralization" for applying XAI to non-layered models. This raises two pertinent questions: How can we effectively balance the trade-offs between model complexity and explainability in practical applications? What are the main challenges in extending the concept of neuralization to more complex machine learning models beyond simpler algorithms like k-means or one-class SVMs?

[Brian-W00]

What is the concept of "neuralization" and how does it help transfer XAI methods developed for neural network classifiers to other models and tasks?

[MarkValadez]

In the concepts methods and new developments, there is a presentation of several visual and numerical tools and methods to provide certain explanatory metrics and media. What I am left wondering about is how do we translate that into the context of non-technical stakeholders and ethically contingent decision making. In other words, something like a representation mapping of the pool-table would be analogous as a black and white square in the prosecution of an individual implying that "a black/white person has a higher propensity for violence and or being guilty in this context." While this would be explanatory, the moral-notions against racial bias should press us to challenge this perfectly explainable but still irksome reason for judgement and explanation.

[mingxuan-he]

While explanability provide many benefits, it might also pose risks since it's generally easier to exploit an explanable model compared to a black-box model, such as targeting the variables with high attribution to model prediction. Some examples include ML use cases in finance/credit industry and crowd-sourced governance. How can the model developer create explanations while ensuring knowledge on these attributions do not result in exploitation?

[kangyic]

For the reading this week, I'm especially interested in XAI algorithms. What are the tasks Perturbation-Based XAI Methods are good at dealing with? How can perturbation-based XAI Methods enhance model trustworthiness? Any limitations?

[hantaoxiao]

In the context of "Explainable Deep Learning: Concepts, Methods, and New Developments," how can we design user-friendly and intuitive visualizations for AI explanations that help non-technical stakeholders grasp complex model behaviors? What innovative visualization techniques could bridge the gap between technical complexity and user understanding?

[anzhichen1999]

In the context of complex models that integrate multiple data modalities, what methods can be employed to ensure these models remain interpretable and explainable? Are there specific techniques that are more effective for certain types of data?

[00ikaros]

What are the main goals and challenges of Explainable AI (XAI) in bringing transparency to complex machine learning models, and how do recent developments address issues such as biases and spurious correlations in training data? Additionally, what are the key concepts and methods in XAI, particularly attribution methods for explaining model decisions? How can explanations be integrated into the training process to improve model efficiency, robustness, and generalization? Lastly, what are the recent advancements in using explanations for pruning, debugging, and improving models, and what is the concept of neuralization in this context? What are the current limitations of explanation methods and potential future research directions in XAI?

[Carolineyx]

I'm curious about the different methodologies for evaluating the quality of explanations in XAI. Given that explanations are inherently subjective and context-dependent, what are the most promising approaches to quantitatively measure the faithfulness, understandability, and applicability of these explanations in deep learning models? How do these evaluation methods account for the diverse requirements of different stakeholders, such as researchers, practitioners, and end-users?

[icarlous]

The “Explainable Deep Learning” chapter highlights the need for transparent AI models, exploring various post hoc explanation techniques and their integration into model training to boost performance and trust. It introduces “neuralization” for non-layered models. How do we balance model complexity and explainability in practice? What challenges arise in applying neuralization to advanced machine learning models?