Implementing SHAP algorithm on visualBERT transformer

[MUZAMMILPERVAIZ]

System Info

Hi @LysandreJik , @NielsRogge, @sgugger, I am working to apply shap algorithm on visualbert. I found a piece of code that run`s well on distilbart-xsum-12-6 , Here is the code:

import numpy as np
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
import shap
import torch
# load transformer language model and tokenizer
tokenizer = AutoTokenizer.from_pretrained("sshleifer/distilbart-xsum-12-6")
model =  AutoModelForSeq2SeqLM.from_pretrained("sshleifer/distilbart-xsum-12-6").cuda()
s=["In this picture, there are four persons: my father, my mother, my brother and my sister."]

explainer = shap.Explainer(model,tokenizer)

shap_values = explainer(s)
![image](https://user-images.githubusercontent.com/69303067/197922659-c1f3cee3-94cb-429d-8123-0920ef0477fd.png)

But i don`t know how to implement the same thing on visualBERT. Is there any repository which demonstrate the implementation of shap algorithm on visualBERT transformer or anyone know how to do this? Thanks for your time.

Who can help?

No response

Information

• [X] The official example scripts
• [X] My own modified scripts

Tasks

• [X] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

from transformers import BertTokenizer, VisualBertForPreTraining, VisualBertForQuestionAnswering
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
#model = VisualBertForPreTraining.from_pretrained('uclanlp/visualbert-nlvr2-coco-pre')
model = VisualBertForQuestionAnswering.from_pretrained("uclanlp/visualbert-vqa")
from datasets import load_dataset
dataset = load_dataset("textvqa")
explainer = shap.Explainer(model,tokenizer)
shap_values = explainer(dataset['train'][0]['question'])

Expected behavior

[sgugger]

You should use the forums for questions like this as we keep issues for bugs and feature requests only.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[NielsRogge]

Closing this issue for the reason above.