Open mrcabbage972 opened 1 year ago
Either tokenize from scratch or convert the arrow files to megatron format
Mayank has a script for converting to megatron, whill share
Either tokenize from scratch or convert the arrow files to megatron format
Mayank has a script for converting to megatron, whill share