add file level FIM and sanity check

bigcode-project / Megatron-LM

Ongoing research training transformer models at scale

Other

371 stars 48 forks source link

Closed loubnabnl closed 11 months ago

loubnabnl commented 11 months ago

This copies @RaymondLi0 's implementation of File level FIM to Megatron

       --fim-rate 0.5 \
       --fim-split-sample \"<file_sep>\" \
       --fragment-fim-rate 0.5 \

I added a dataloader sanity check flag/code from brrr, I can remove it if it's not needed