How to train multiple binariey files at the same time or merge them?

NVIDIA / Megatron-LM

Ongoing research training transformer models at scale

Other

10.08k stars 2.27k forks source link

Open Liangyz2019 opened 2 months ago

Liangyz2019 commented 2 months ago

Suppose I have three datasets and convert to binary files train1.bin, train1.idx, train2.bin, train2.idx, train3.bin, train3.idx.

During training, I want these three data sets to be merged into one and trained together. How should I do this?

It would be better if there are clear examples and guidance, thank you very much.

Arvid-pku commented 2 months ago

You can use blend_per_split in the BlendedMegatronDatasetConfig.

github-actions[bot] commented 4 days ago

Marking as stale. No activity in 60 days.