LysandreJik
commented
3 years ago Hello! Thanks a lot for the well-crafted issue and reproducer, this is very helpful. Regarding your problem 2, I have a question: why are you saving the tokenizer's model, rather than the tokenizer itself?
I would argue that saving the entire tokenizer in a tokenizer.json would be better:
# And now it is ready, we can save the vocabulary with
tokenizer.save('./tok/tokenizer.json')Then you'll be able to reload your fast tokenizer (that is looking for a tokenizer.json file!) seamlessly:
from transformers import BigBirdTokenizerFast
tokenizer = BigBirdTokenizerFast.from_pretrained("tok", max_len=16000)I also verified that you do indeed recover the same encoding as when using the tokenizers library:
>>> tokenizer("23 39999 999 8888 212").tokens()
['23', '39999', '999', '8888', '212']Regarding your first question, I don't see anywhere in your code where you're adding a BERT template processor. I've taken the liberty to add it right after your tokenizer creation, see below. I am unaware of the error you got, but when trying it I had an error saying that tokenizer.token_to_id("<s>") was returning None.
To fix this you can specify that <s> and <s/> are special tokens when initializing your BPE trainer, as I have done below.
from tokenizers import Tokenizer, trainers
from tokenizers.models import BPE
from tokenizers.normalizers import Lowercase
from tokenizers.pre_tokenizers import CharDelimiterSplit
# We build our custom tokenizer:
tokenizer = Tokenizer(BPE())
tokenizer.normalizer = Lowercase()
tokenizer.pre_tokenizer = CharDelimiterSplit(' ')
# We can train this tokenizer by giving it a list of path to text files:
trainer = trainers.BpeTrainer(special_tokens=["[UNK]", "<s>", "</s>"], show_progress=True)
tokenizer.train(files=['/content/dataset.txt'], trainer=trainer)
from tokenizers.processors import BertProcessing
import tokenizers
tokenizer.post_processor = tokenizers.processors.BertProcessing(
("</s>", tokenizer.token_to_id("</s>")),
("<s>", tokenizer.token_to_id("<s>")),
)
tokenizer.enable_truncation(max_length=16000)After this, encoding a sequence returns the correct tokens with the correct special tokens:
>>> tokenizer.encode("23 39999 999 8888 212").tokens
['<s>', '23', '39999', '999', '8888', '212', '</s>']
neel04
Environment info
transformersversion: 4.6.1Who can help
Information
Model I am using (Bert, XLNet ...):
BigBirdThe problem arises when using:
The tasks I am working on is:
To reproduce
Steps to reproduce the behavior:
I have a dataset for which I wanted to use a tokenizer based on whitespace rather than any subword segmentation approach.
This snippet I got off github has a way to construct and use the custom tokenizer that operates on whitespaces:-
I wanted to use it for pre-training the
BigBirdmodel, but facing two issues:tokenizerabove to convert tokenized sentences in model-friendly sequencesThis returns me an error, and without any preprocessing the output does not contain the sequence start and end tokens (
<s>;</s>) as expected.since it yields the error that my directory does not ‘reference’ the tokenizer files, which shouldn’t be an issue since using
RobertaTokenizerFastdoes work - I assume it has something to do in the tokenizationpost-processingphase.Fully Reproducible Colab
I am really confused about this - I have created a fully reproducible colab notebook, with commented problems and synthetic data. Please find it here.
Thanx a ton in advance!!