Open allanj opened 1 year ago
I saw this function, but never called
https://github.com/paperswithcode/galai/blob/3a724f562af1a0c8ff97a096c5fbebe579e2160f/galai/model.py#L143-L172
But quite confused that it seems we never use the bos and eos tokens,
even in transformer, they are None if we print out the bos, eos, pad tokens.
None
I'm quite confused about bos and eos, do we actually use them or not
I saw this function, but never called
https://github.com/paperswithcode/galai/blob/3a724f562af1a0c8ff97a096c5fbebe579e2160f/galai/model.py#L143-L172
But quite confused that it seems we never use the bos and eos tokens,
even in transformer, they are
None
if we print out the bos, eos, pad tokens.I'm quite confused about bos and eos, do we actually use them or not