search

cdpierse / transformers-interpret

Model explainability that works seamlessly with 🤗 transformers. Explain your transformers model in just 2 lines of code.
Apache License 2.0
1.29k stars 97 forks source link

Support for Longformer #130

Open mpafla opened 1 year ago

mpafla commented 1 year ago

I am trying to get word attributions running for the Longformer model. 

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
from transformers_interpret import QuestionAnsweringExplainer

question = "How many programming languages does BLOOM support?"
context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."

tokenizer = AutoTokenizer.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")
model = AutoModelForQuestionAnswering.from_pretrained("valhalla/longformer-base-4096-finetuned-squadv1")

qa_explainer = QuestionAnsweringExplainer(
    model,
    tokenizer,
)

word_attributions = qa_explainer(
    question,
    context,
)

print(word_attributions)
print(qa_explainer.predicted_answer)
qa_explainer.visualize("bert_qa_viz.html")

With this code I get the following error. AssertionError: There should be exactly three separator tokens: 2 in every sample for questions answering. You might also consider to set `global_attention_mask` manually in the forward function to avoid this error.

Is this expected? How do I get the word attributions for the Longformer model (if I can)?