Multi node training - Githubissues

allenai / OLMo

Modeling, training, eval, and inference code for OLMo

https://allenai.org/olmo

Apache License 2.0

4.2k stars 392 forks source link

Multi node training #640

Open shahizat opened 5 days ago

shahizat commented 5 days ago

❓ The question

Dear all,

Could you please suggest what parameter(s) increase or change when conducting multi-node training of OLMo, so that we can observe the difference between single-node training and calculate the network overhead? throughput -tokens per second?

Thank you in advance