🚀 Efficiently (pre)training foundation models with native PyTorch features, including FSDP for training and SDPA implementation of Flash attention v2.
Previous BOS/EOS support occurred at the line level, as in encoder-decoder training. With Llama3 BOS/EOS usage we want to extend this behavior to the document level. This PR adds support for bos_token flag in training and dataloading. A new unit test ensures that chunking behavior remains consistent and correct.
Previous BOS/EOS support occurred at the line level, as in encoder-decoder training. With Llama3 BOS/EOS usage we want to extend this behavior to the document level. This PR adds support for
bos_tokenflag in training and dataloading. A new unit test ensures that chunking behavior remains consistent and correct.