Add github dataset from redpajama - Githubissues

kojimano / Megatron-DeepSpeed-ABCI

Other

5 stars 2 forks source link

Add github dataset from redpajama #26

Open kuriyan1204 opened 1 year ago

kuriyan1204 commented 1 year ago

[ ] download script
[ ] preprocess
[ ] binarize for megatoron

kuriyan1204 commented 1 year ago

note : need extra preprocess other than merging file?