issues
search
bigcode-project
/
Megatron-LM
Ongoing research training transformer models at scale
Other
376
stars
49
forks
source link
WIP: Fim
#9
Closed
RaymondLi0
closed
2 years ago
RaymondLi0
commented
2 years ago
Fill-in-the-middle: #2 , #8
Add code from
https://github.com/EleutherAI/gpt-neox/blob/FIM-clean/megatron/data/gpt2_dataset.py
, with some changes
TODO:
[x] Add sentinel tokens to the tokenizer
[x] add SPM+PSM mode
[ ] Check that it does not hurt training speed
Fill-in-the-middle: #2 , #8
TODO: