Closed alshahrani2030 closed 4 years ago
Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.
Are you importing the versions of BertTokenizer/BertForSequenceClassification that are in bertviz.pytorch_transformers_attn? This is a forked version of the transformers library specifically for bertviz.
Hi Thank you for the quick reply. yes I did, here is the screenshots
Have you found a solution? I have a different error when trying to load a BERT fine tuned classification model to analyze via bertviz:
from bertviz.pytorch_transformers_attn import BertForSequenceClassification
from bertviz.pytorch_transformers_attn import BertModel, BertTokenizer
from bertviz.head_view import show
%%javascript
require.config({
paths: {
d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
}
});
import torch
from pytorch_pretrained_bert import BertModel, BertTokenizer, modeling
from pytorch_pretrained_bert.file_utils import PYTORCH_PRETRAINED_BERT_CACHE
from pathlib import Path
save_dir = Path('/home/bert/savedmodelfolder/')
save_dir.mkdir(exist_ok=True)
bert_model = 'bert-base-cased'
save_file = save_dir / modeling.WEIGHTS_NAME
config_file = save_dir / modeling.CONFIG_NAME
model_base = BertModel.from_pretrained(
bert_model,
cache_dir=PYTORCH_PRETRAINED_BERT_CACHE / 'distributed_{}'.format(-1)
)
model_type = 'bert'
model_version = 'bert-base-cased'
do_lower_case = False
# Loading
model_state_dict = torch.load(save_file)
state_dict_with_prefix = {}
for key, value in model_state_dict.items():
state_dict_with_prefix['bert.' + key] = value
model_loaded = BertModel.from_pretrained(save_dir, state_dict=state_dict_with_prefix)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"
show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)
The error is:
AttributeError Traceback (most recent call last)
<ipython-input-15-a0212be47ce3> in <module>
30 sentence_a = "The cat sat on the mat"
31 sentence_b = "The cat lay on the rug"
---> 32 show(model_loaded, model_type, tokenizer, sentence_a, sentence_b)
~/bertviz/head_view.py in show(model, model_type, tokenizer, sentence_a, sentence_b)
57 os.path.join(os.getcwd(), os.path.dirname(__file__)))
58 vis_js = open(os.path.join(__location__, 'head_view.js')).read()
---> 59 attn_data = get_attention(model, model_type, tokenizer, sentence_a, sentence_b)
60 params = {
61 'attention': attn_data,
~/bertviz/attention.py in get_attention(model, model_type, tokenizer, sentence_a, sentence_b, include_queries_and_keys)
57 else:
58 if model_type == 'bert':
---> 59 tokens_a = [tokenizer.cls_token] + tokenizer.tokenize(sentence_a) + [tokenizer.sep_token]
60 tokens_b = tokenizer.tokenize(sentence_b) + [tokenizer.sep_token]
61 token_type_ids = torch.LongTensor([[0] * len(tokens_a) + [1] * len(tokens_b)])
AttributeError: 'BertTokenizer' object has no attribute 'cls_token'
The tokenizer seems to no know what BERT's CLS tokens are... Any ideas? Thank you.
Hmm, that's an odd one. When I look at the version of BertTokenizer on github, it inherits from tokenization_utils.PretrainedTransformer, which has a cls_token method. Is it possible that somehow the code in either tokenization_bert or tokenization_utils (both in pytorch_transformers_attn library) are out of date? Or somehow being imported from a different path? BTW, I'm working on a change that will decouple bertviz from the transformers library, so this would likely resolve these sorts of issues. I hope to finish it in the next week or two. Thanks.
Ah yes, I found the problem. Thanks! Everything is beautiful at your end, the paths were incorrect. Thanks again for this tool.
Thanks for letting me know. BTW, BertViz is now decoupled from transformers for the head and model views, so there shouldn't be a risk of these issues in the future.
That was fast! Hopefully transformers will stay stable and nothing will break in the future. They tend to keep changing how things work.
I now can visualize my tuned classification model, but interpreting it is hard in my case. It seems that all the attention heads generally attend equally to all words, across all layers. The pattern is always the same as this:
Is there a simple way to inspect the actual attention scores? If not, I'll probably just modify it so that the numbers appear as well.
Again, many many thanks for this.
Hmm, this is a problem that has happened with other people who loaded fine-tuned models on the previous version of BertViz, e.g., https://github.com/jessevig/bertviz/issues/12 . They were able to resolve this issue but the cause was never clear. I would recommend trying the new version of bertviz in this case. Let me know if any of these solutions work for you.
Thanks. Ok, I'm trying to take the code in #12 and update it to the most recent bertviz version. Here's what I got so far (notice the UnicodeDecodeError error when saving the tuned model):
from bertviz import head_view
from transformers import BertTokenizer, BertForSequenceClassification
%%javascript
require.config({
paths: {
d3: '//cdnjs.cloudflare.com/ajax/libs/d3/3.4.8/d3.min',
jquery: '//ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min',
}
});
model_path = '/bert/outputs/'
save_model = BertForSequenceClassification.from_pretrained('/bert/outputs/pytorch_model.bin', num_labels=2)
torch.save(save_model.state_dict(), model_path)
from bertviz.transformers_attn import BertModel, BertTokenizer
from bertviz.head_view_bert import show
from bertviz.transformers_attn import BertForSequenceClassification
model_version = 'bert-base-cased'
do_lower_case = True
model_state_dict = torch.load(model_path)
assert model_state_dict is not None
model = BertModel.from_pretrained(model_version, state_dict=model_state_dict,output_attentions=True,num_labels=2)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case)
sentence_a = "The cat sat on the mat"
sentence_b = "The cat lay on the rug"
inputs = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt', add_special_tokens=True)
token_type_ids = inputs['token_type_ids']
input_ids = inputs['input_ids']
attention = model(input_ids, token_type_ids=token_type_ids)[-1]
input_id_list = input_ids[0].tolist() # Batch index 0
tokens = tokenizer.convert_ids_to_tokens(input_id_list)
head_view(attention, tokens)
Error:
I1125 08:20:57.783921 140693827761984 configuration_utils.py:148] loading configuration file /bert//outputs/paraphrase/pytorch_model.bin
---------------------------------------------------------------------------
UnicodeDecodeError Traceback (most recent call last)
<ipython-input-5-b4e6db36dc04> in <module>
1 model_path = '/home/rpc/DeepL/Paraphrase/outputs/paraphrase/'
----> 2 save_model = BertForSequenceClassification.from_pretrained('/outputs/pytorch_model.bin', num_labels=2)
3 torch.save(save_model.state_dict(), model_path)
4
5 from bertviz.transformers_attn import BertModel, BertTokenizer
~/.local/lib/python3.6/site-packages/transformers/modeling_utils.py in from_pretrained(cls, pretrained_model_name_or_path, *model_args, **kwargs)
285 cache_dir=cache_dir, return_unused_kwargs=True,
286 force_download=force_download,
--> 287 **kwargs
288 )
289 else:
~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
152
153 # Load config
--> 154 config = cls.from_json_file(resolved_config_file)
155
156 if hasattr(config, 'pruned_heads'):
~/.local/lib/python3.6/site-packages/transformers/configuration_utils.py in from_json_file(cls, json_file)
184 """Constructs a `BertConfig` from a json file of parameters."""
185 with open(json_file, "r", encoding='utf-8') as reader:
--> 186 text = reader.read()
187 return cls.from_dict(json.loads(text))
188
/usr/lib/python3.6/codecs.py in decode(self, input, final)
319 # decode input (taking the buffer into account)
320 data = self.buffer + input
--> 321 (result, consumed) = self._buffer_decode(data, self.errors, final)
322 # keep undecoded input until the next call
323 self.buffer = data[consumed:]
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte
Thanks for following up. A couple of thoughts:
Thanks again for the quick reply. This helped. You were right, the extra calls were necessary and there were problems with paths again. I got it to work, but I'd like to see the "Attention" dropbox menu Sentence_A -> Sentence_B menu. How do I bring it back? Thank you for your patience.
Hi Rui, I neglected to mention that as part of that change, I removed the functionality to filter the attention, because it was harder to implement when using transformers directly. I do plan to add the functionality back soon, but I don't have an exact timeline unfortunately.
I see. I found that view quite intuitive (especially when examining longer strings). I'll be on the lookout for it's return. Cheers!
Hi, I've created a new branch which has the Attention dropdown: https://github.com/jessevig/bertviz/tree/sentence_pair_filter . See head_view_bert.ipynb. Please give it a try!
Fantastic. This works nicely. Much appreciated!
Good to hear! BTW, it's in master now too.
'bert-base-cased'
我也是这个问题 请问是哪个路径的问题?
Hi, Thank you for this great work. can I use this code to plot my model(I am useing BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
model_type = 'bert' model_version = 'bert-base-uncased' do_lower_case = True model = model #(this my model)
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased', do_lower_case=True)
tokenizer = BertTokenizer.from_pretrained(model_version, do_lower_case=do_lower_case) sentence_a = sentences[0] sentence_b = sentences[1] call_html() show(model, model_type, tokenizer, sentence_a, sentence_b) I changed only the model with my model, and the sentences and I got this error??!!please help or share any blog that explain how to plot my model AttributeError: 'BertTokenizer' object has no attribute 'cls_token'
Thank you in advance