Integrated gradients for nlp models. (Explainable AI)

keras-team / keras-hub

Pretrained model hub for Keras 3

Apache License 2.0

763 stars 230 forks source link

Integrated gradients for nlp models. (Explainable AI) #93

Closed innat closed 1 year ago

innat commented 2 years ago

Describe feature The Integrated Gradients (IG) can be a great tool to understand the neural network.

How API will change? It would enhance as follows:


from keras_nlp.utils.visualization import IntegratedGradients as IG 

ig  = IG (model, layer=layer, ...)

Candidate Solution

ALIBI-Explain: It's a toolbox that provides many solutions, including IG in TensorFlow 2 (Keras).

Demo o/p:

ig.explain(sample)

mattdangerw commented 2 years ago

Definitely, it would be great to start adding explainability tools here! I was actually just wishing we had precisely this while working on a guide for keras-nlp.

Would you be interested in contributing?

Also, trying to understand the design proposed more...

IG is it's own non-layer class?
What assumptions are being made about the model parameter passed in? If any.

abheesht17 commented 2 years ago

Hey, @innat! Are you still working on this? If not, I'd like to work on this. Thanks! :)

innat commented 2 years ago

@abheesht17 I'm not working on it. Please, take action if you want.

abheesht17 commented 2 years ago

Thanks, @innat! @mattdangerw, I can take this up (if we want this layer) :)