When flag --ignore_prefix is set, the positions for the positional encoding will be centered at the separator token. Any positions left of the separator token will get negative positions.
Example:
vanilla:
V SG 3 PRS u s e
0 1 2 3 4 5 6 7
separator-centered:
V SG 3 PRS u s e
-4 -3 -2 -1 0 1 2 3
When flag
--ignore_prefixis set, the positions for the positional encoding will be centered at the separator token. Any positions left of the separator token will get negative positions.Example:
vanilla: V SG 3 PRS u s e
0 1 2 3 4 5 6 7
separator-centered: V SG 3 PRS u s e
-4 -3 -2 -1 0 1 2 3