Do we ever use the start and end tokens - Githubissues

paperswithcode / galai

Model API for GALACTICA

Apache License 2.0

2.68k stars 276 forks source link

Do we ever use the start and end tokens #77

Open allanj opened 1 year ago

allanj commented 1 year ago

I saw this function, but never called

https://github.com/paperswithcode/galai/blob/3a724f562af1a0c8ff97a096c5fbebe579e2160f/galai/model.py#L143-L172

But quite confused that it seems we never use the bos and eos tokens,

even in transformer, they are None if we print out the bos, eos, pad tokens.

I'm quite confused about bos and eos, do we actually use them or not