GPT2

[BigSalmon2]

I aplogize for asking another question, but I was curious to know if it was possible to involve GPT2? I am aware that gltr.io did something of the sort, but I am not so sure how to get that into heatmap form on a colab notebook.

[AndreasMadsen]

Did you look at the HuggingFace example: https://colab.research.google.com/github/AndreasMadsen/python-textualheatmap/blob/master/notebooks/huggingface_bert_example.ipynb

[BigSalmon2]

Did you look at the HuggingFace example: https://colab.research.google.com/github/AndreasMadsen/python-textualheatmap/blob/master/notebooks/huggingface_bert_example.ipynb

Yes, and I will assume it does not. There is no [MASK] token in GPT2 like BERT. I’ll try it out, though.

[AndreasMadsen]

Yes. GPT2 is a next token predictor model. In those terms it is more similar to the RNN model in my distill.pub paper. But you will be able to use the huggingface example to see how the gradent w.r.t. the input can be computed.

[BigSalmon2]

Thank you for your response. I hope I am not wasting too much of your time.

text = ("""In an act of [MASK], I hurried down the staircase, where I would find myself [MASK] the front-door. """)

from transformers import GPT2Tokenizer, GPT2LMHeadModel

model = GPT2LMHeadModel.from_pretrained('gpt2') 
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
text_index = tokenizer.encode('man',add_prefix_space=True)
vector = model.transformer.wte.weight[text_index,:]

from textualheatmap import TextualHeatmap 
heatmap = TextualHeatmap(facet_titles = ['GPT2'], show_meta=True, width = 1000)
heatmap.set_data([
    compute_textual_saliency(model, vector, tokenizer, text),
])

I tried doing this, but got this error: 'Tensor' object has no attribute 'get_shape.'