Closed josepsmartinez closed 4 years ago
If a word contains the special character º or ª, its token is decoded as [UNK] (unknown).
º
ª
[UNK]
Code that demonstrates the issue:
from transformers import AutoTokenizer model_dir = "models/portuguese-bert-lener_br" tokenizer = AutoTokenizer.from_pretrained(model_dir) def encode_decode(t): return tokenizer.decode(tokenizer.encode(t)) print(encode_decode("nº 396")) print(encode_decode("n 396")) print(encode_decode("Srª Maria")) print(encode_decode("Sr Maria"))
Is there anything I am doing wrong or is the model incapable of embedding words containing such characters?
Hi jotapem,
Sorry for the delayed response. Yes, unfortunately these symbols were not included in the generated WordPiece vocabulary.
Thank you for the answer.
If a word contains the special character
º
orª
, its token is decoded as[UNK]
(unknown).Code that demonstrates the issue:
Is there anything I am doing wrong or is the model incapable of embedding words containing such characters?