reviewer-3-comment-3

[docsteveharris]

3. Recent work have questioned the value of model explainability, particularly because many explainability methods exhibit flaws. See Ghassemi, Oakden-Rayner, and Beam 2021 and https://arxiv.org/abs/2202.01602, as well as references within. I think the recommendation for model explainability should be qualified with this fact that the current explainability methods leave much to be desired.

[Waty-Lilaonitkul]

Hi Steve, Just sent an email to you on the explainability methods section with

edited text for latex doc added reference for latex doc

[docsteveharris]

We argue that model explainability methods [@gunning2019],[@mueller2019],[@vilone2020],[@Linardatos2020] need to be prioritised to help systematise and coordinate the processes of model troubleshooting by developers, risk-management by AI-backed service provider and system-checks by auditors. Most AI models that operate as ‘black-box models’ are unsuitable for mission-critical domains, such as healthcare, because they pose risk scenarios where problems that occur can remain masked and therefore undetectable and unfixable. As highlighted in [@Ghassemi 2021],[@Satyapriya2022] explainability methods cannot yet be relied on to provide a determinate answer as to whether an AI-recommendation is correct. However, explainability methods that highlight decision-relevant parts of AI representations and for measuring and benchmarking interpretability [@Doshi-Velez2017],[@Hoffman2018] are particularly promising for risk management as they can be used to structure a systematic interrogation of the trade-off between interpretability, model accuracy and the risk of model misbehaviour.