Closed JanKaul closed 2 years ago
@JanKaul That's actually true in my experimentation, I don't know why that is the case. Maybe in a little future I could train the spm model myself from the source.
I might have an idea why that's the case. You have to distinguish between user defined symbols and control symbols. According to the documentation:
I think [CLS]
and [SEP]
are added as control symbols and have to be added manually after encoding.
I'm trying to use albert for a question answering task. Therefore I want to encode my input text with sentencepiece to use it as input for the albert model. I initialize the sentencepiece model by loading the model file from one of the official tar files. The encoding seems to work fine except for the control symbols. The input
[CLS]
gets encoded into three pieces. The expected behavior would be to get one piece. The same happens for[SEP]
.Here is an example:
Ouptut:
'▁[', 'CLS', ']'
Am I doing something wrong? Is there a way to specify the control symbols without training the model? I would like to avoid training the model every time I load the model. I would really appreciate your help. Thank you