Pretraining dataset and code request

myshell-ai / JetMoE

Reaching LLaMA2 Performance with 0.1M Dollars

Apache License 2.0

937 stars 72 forks source link

Pretraining dataset and code request #6

Open hitalex opened 2 months ago

hitalex commented 2 months ago

Will the pretraining datasets and corresponding code be open-sourced?

Thanks!

tranhd95 commented 2 months ago

Hi, thanks for the great work. I'd also be interested particularly in training code or at least if you can share some multi-node settings details. What tech did you use for parallelization across GPU nodes?

lqniunjunlper commented 2 months ago

Same. Looking forward to the open-source training code and details.

shamanez commented 2 months ago

Thanks for the amazing work and the paper. Really would love to explore your training code.

geronimi73 commented 2 months ago

Zengyi-Qin commented 2 months ago

https://huggingface.co/jetmoe/jetmoe-8b/discussions/5