search

bigcode-project / Megatron-LM

Ongoing research training transformer models at scale
Other
376 stars 49 forks source link

add multi-validation for gpt training #32

Closed RaymondLi0 closed 1 year ago

RaymondLi0 commented 1 year ago

Adding code from: https://github.com/bigscience-workshop/Megatron-DeepSpeed/pull/97

TODO: improve loading the data-paths from a file to read a structured format (json?yaml?) instead of parsing a text file. This could be in another PR