Closed jandono closed 2 years ago
Some additional info that might help. Encodings looks like: encodings = [batch_encoding_1, ... , batch_encoding_2]
. Each batch encoding looks like:
{'input_ids': tensor([[ 101, 1006, 1039, 1007, 2065, 1996, 13666, 11896, 2000, 14037,
2007, 2019, 14987, 2104, 2023, 11075, 3429, 1010, 1999, 2804,
2000, 2151, 2060, 2128, 7583, 3111, 1997, 1996, 4054, 1010,
1996, 4054, 2089, 4685, 2008, 14987, 2006, 1996, 13666, 1005,
1055, 6852, 1998, 2151, 3465, 22667, 2011, 1996, 4054, 2097,
2022, 1037, 7016, 2349, 2013, 1996, 13666, 2000, 1996, 4054,
1012, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
And this is the line that eventually raises an exception:
I managed to make a small reproducible example:
from transformers import BertTokenizer
from torch import tensor
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
encodings = [{'input_ids': tensor([[ 101, 1006, 1039, 1007, 2065, 1996, 13666, 11896, 2000, 14037,
2007, 2019, 14987, 2104, 2023, 11075, 3429, 1010, 1999, 2804,
2000, 2151, 2060, 2128, 7583, 3111, 1997, 1996, 4054, 1010,
1996, 4054, 2089, 4685, 2008, 14987, 2006, 1996, 13666, 1005,
1055, 6852, 1998, 2151, 3465, 22667, 2011, 1996, 4054, 2097,
2022, 1037, 7016, 2349, 2013, 1996, 13666, 2000, 1996, 4054,
1012, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}, {'input_ids': tensor([[ 101, 1006, 1037, 1007, 2202, 2046, 4070, 2035, 1997, 1996, 7882, 6214,
1997, 1996, 3563, 3105, 1010, 2164, 1996, 3872, 2030, 3635, 1997, 1996,
7170, 2000, 2022, 2333, 1010, 3292, 2000, 2022, 7837, 2005, 4651, 1010,
2334, 4026, 3785, 1010, 2051, 1997, 2154, 1010, 3517, 3403, 2335, 1010,
2569, 2609, 3785, 1998, 2060, 2569, 6214, 1025, 1998, 102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}]
batched_encodings = tokenizer.pad(encodings, padding='longest', return_tensors='pt')
@LysandreJik any update on this?
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
@LysandreJik @patrickvonplaten @sgugger
I apologize for tagging Patric and Sylvain, but as Lysandre seems to be busy, do you perhaps know someone who can help with this?
The tokenizer.pad
method only applies padding for list of examples, so each of the elements in your encoding
should be one-dimensional. If you remove the extra pair of [] in all your tensors in your minimal example, it will work.
Also please use the forums for questions around the library as we keep the issues for bugs and feature requests only.
Thanks a lot @sgugger , I posted it her as it looked like a bug to me based on the documentation. Additionally, those extra set of parenthesis come from the tokenizer not me. When running:
encoding = self.tokenizer(text,
add_special_tokens=True,
max_length=512,
padding=False,
truncation=True,
return_attention_mask=True,
return_tensors='pt')
You get those extra parenthesis. I am assuming they come because in the background of the __call__
method, batch_encode
is called instead of encode
. Am I doing something wrong in the way I am using the tokenizer? My main goal is to simply tokenize the entire dataset beforehand, and only pad during training.
You should not use return_tensors='pt'
for just one text, that option is designed to create batches you directly pass to your model. So if you use it with one text, you get a batch of one encoding. Either add [0] to select the only element of that batch in your dataset, or create the tensors in the collate function.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Environment info
transformers
version: 4.2.2Who can help
@LysandreJik
Information
Model I am using (Bert, XLNet ...): Bert
The problem arises when using:
The tasks I am working on is:
To reproduce
I am trying to avoid tokenizing in the collator in order to improve the speed of Data Loading, which is why I wanted to tokenize everything in advance and then simply pad in the collator. I can't provide the entire code however here are my
Dataset
and myCollator
which will hopefully be enough.Error:
Full error message:
Expected behavior
I would expect
self.tokenizer.pad(encodings, ... )
in the collator to work without issues when given a list ofBatchEncoding
elements.