Patch Description
Add support for training with marmot data
Testing steps
Conduct ablation study for marmot on RSC. log: /checkpoint/vllm/cm3leon/experiments/ablations/DATA_ABLATION_marmot2_760m/DATA_ABLATION_marmot2_760m.bm_none.fp16.bf16.trunc.fmis0.0.sig0.006.pos0.0002.nobias.noaffln.relu.transformer_lm_megatron.nlay24.emb1536.lrnpos.0emb_scale.tps8192.adam.b2_0.95.cl1.0.lr0.00025.wu1500.dr0.1.atdr0.0.0emb_dr.wd0.1.ms8.mu119209.s1.ngpu256.
Patch Description Add support for training with marmot data
Testing steps Conduct ablation study for marmot on RSC. log: /checkpoint/vllm/cm3leon/experiments/ablations/DATA_ABLATION_marmot2_760m/DATA_ABLATION_marmot2_760m.bm_none.fp16.bf16.trunc.fmis0.0.sig0.006.pos0.0002.nobias.noaffln.relu.transformer_lm_megatron.nlay24.emb1536.lrnpos.0emb_scale.tps8192.adam.b2_0.95.cl1.0.lr0.00025.wu1500.dr0.1.atdr0.0.0emb_dr.wd0.1.ms8.mu119209.s1.ngpu256.