search

pytorch / captum

Model interpretability and understanding for PyTorch
https://captum.ai
BSD 3-Clause "New" or "Revised" License
4.8k stars 485 forks source link

Could Captum be used for sequence tagging problems? #374

Open p16i opened 4 years ago

p16i commented 4 years ago

Hi,

I'm currently working on a sequence tagging problem, and I'm interested in investigating what tokens in the sequence are relevant for each step's predicting.

NarineK commented 4 years ago

Hi @heytitle , you should be able to use captum for sequence tagging. We don't have a tutorial on that but I'd recommend to start from IG and attribute predicted output at each step to the input embeddings. Here is an older example on translation. https://github.com/pytorch/captum/blob/4378c1c7ae733a56fa781881faa3cf59b8829b6b/tutorials/seq2seq_interpret.ipynb

p16i commented 4 years ago

@NarineK Thanks for the reply!

Just out of my curiosity, what don't we have that tutorial on the website?

NarineK commented 4 years ago

It was long time ago. I didn't get time to review it. Seq2seq models can also be more computationally expensive but we can definitely think of adding a tutorial. Do you have any recommendations on seq2seq models in PyTorch that would be good to have a tutorial on?

p16i commented 4 years ago

I'll take a look and let you know if I come up with something. The problem that I'm currently working is about word segmentation for Thai language. It's quite specific and might not be a good example to illustrate Captum's features.

LopezGG commented 4 years ago

@NarineK : I used your tutorial to create one for seq2seq model (i.e NER) here . Could you please review it ?

lipka-clazzpl commented 4 years ago

Hi @LopezGG, great tutorial,

We also find that the last few layers of the target word "Rob" also has a strong positive influence.

Actually last few layers (starting with 1 up to layer 4) are quite cold, think You wanted to point out to layers 5-7.

LopezGG commented 4 years ago

@lipka-clazzpl : Thanks for your feedback. I believe the lighter color refers to positive feedback (0.8-1.0) based on the scale here. Am I missing something image

lipka-clazzpl commented 4 years ago

@LopezGG correct, lighter color=more positive feedback, thought You wanted to explain attribution for word "Rob" solely with all the layers, my misinterpretation, You clearly explain "Rob" with "win", ",", "in" and "surprise" at lowest layers.