search

huggingface / nanotron

Minimalistic large language model 3D-parallelism training
Apache License 2.0
1.14k stars 107 forks source link

[Refactor] Add minimal ParallelContext #16

Closed xrsrke closed 8 months ago

xrsrke commented 8 months ago

Only add ParallelContext with parallel_context.[dp_pg|tp_pg|pp_pg] as our last discussion [link]

And it's passed and failed all the tests that it did before the refactoring.

image