Improve loading of the data-paths

bigcode-project / Megatron-LM

Ongoing research training transformer models at scale

Other

371 stars 48 forks source link

Improve loading of the data-paths #38

Open RaymondLi0 opened 1 year ago

RaymondLi0 commented 1 year ago

The --(train|valid|test)-weighted-split-paths-path arguments (added in #32 ) parses the data arguments from a file in a specific format. Loading could be made simpler by reading a structured file (json or yaml). Such file would be more human-readable too.