This PR adds an example of finetuning a GPT2 model using veScale API in 4D parallelism: Data, Tensor, Sequence, and Optimizer Parallelism. There are near-zero changes in the model code. In addition, this PR also improves factory methods for DTensors and simplifies DModule APIs.
This PR adds an example of finetuning a GPT2 model using veScale API in 4D parallelism: Data, Tensor, Sequence, and Optimizer Parallelism. There are near-zero changes in the model code. In addition, this PR also improves factory methods for DTensors and simplifies DModule APIs.