Closed nooralahzadeh closed 4 years ago
By default, SentencePieceTrainer
assigns ID 1 as <s>
and ID 2 as </s>
. Check here
I prefer [SOS]
and [EOS]
in text instead of <s>
and </s>
, so I passed my custom symbols as --control-symbols
. Internally, SentencePieceTrainer
reserves ID 0 for <unk>
(which cannot be changed as far as I know), and other control symbols are assigned from ID 3 (in presence of default <s>
and </s>
) or ID 1 (in absence of <s>
and </s>
).
So I turned off default <s>
and </s>
, and instead provided [SOS]
and [EOS]
so they get ID 1 and 2 respectively.
Edited title for others to search easily. :-)
Thanks. How about [PAD]'s id, is it zero by default?
[PAD]
and <unk>
are same — we use the same token to right-pad captions and represent out-of-vocabulary tokens, similar to recent image captioning models. It is ID 0 by default. As far as I remember, I always refer its token as <unk>
, and the corresponding variable name in code is padding_idx
.
Hi, I saw that in SentencePieceTrainer, as below you made EOS and BOS and MASK and PAS tokens equal to Zero
" --bos_id=-1 --eos_id=-1" " --control_symbols=[SOS],[EOS],[MASK]"
However, during the captioning, you definesos_index: int = 1, eos_index: int = 2,
I am wondering if these setups , have any effects?