Open xrsrke opened 10 months ago
I think I'll do this tonight since it seems the easiest
@isamu-isozaki Awesome, thank you! I will get back to you in a few hours with all the details!!
@xrsrke I was thinking of maybe just inheriting from transformer's Trainer. wdyt?
@isamu-isozaki Nope, I just checked Trainer
from transformers
. They modified our model's devices and stuff. We prefer implementing our own so we can incorporate distributed logging and callback in a specific rank, ParallelMode... and future changes. I just added some demo code (link).
Also one note, we only apply a specific parallel mode based on the parallel_context. For example, if data_parallel_size is greater than 1, then we wrap the model with DataParallel.
Notes
DataParallel
,TensorParallel
andPipelineParallel
modules. The user just plugs in their model and dataloader and trains. Similar totransformers
.APIs
Trainer
Trainer Callback
DistributedDataLoader
TODOs