search

HomebrewNLP / Olmax

HomebrewNLP in JAX flavour for maintable TPU-Training
BSD 2-Clause "Simplified" License
45 stars 6 forks source link

Multi-Host Scaling #40

Closed ClashLuke closed 2 years ago

ClashLuke commented 2 years ago

closes #13

ClashLuke commented 2 years ago

Code works as seen in the run on wandb. Next, we should discuss whether we have number_of_hosts runs on wandb, but I couldn't figure out how to synchronise wandb configs yet.