Closed tbenthompson closed 11 months ago
cc @ArthurZucker
You should try with tokenizer.decode(ids, clean_up_tokenization_spaces = False)
:
>>> import transformers
>>> tokenizer = transformers.AutoTokenizer.from_pretrained("EleutherAI/pythia-12b-deduped")
>>> text = " there is this someone 'you' who has the ability of 'sensing things"
>>> ids = tokenizer.encode(text)
>>> print(repr(tokenizer.decode(ids, clean_up_tokenization_spaces = False)))
" there is this someone 'you' who has the ability of 'sensing things"
Ok, thanks! For my own understanding, why is the default clean_up_tokenization_spaces = True
? Without that setting, decode
and encode
are much closer to being the inverse of each other. Intuitively, that seems like it should be the default goal of decode
Good question, has been that way for a long time, I think this was to reflect some of our original tokenizers that were adding spaces. I'll check if we can safely remove this / set to False
by default!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
System Info
transformers
version: 4.32.1Reproduction
When running this code:
I get the output:
Expected behavior
The first string produced by
tokenizer.decode
is an incorrect decoding. The second string frombatch_decode
is correct. The first string is missing a space before'sensing
.