Open yzxing87 opened 2 years ago
If you pickle the model for single GPU, anything will be fine because AllToAll is not included in Tutel's MoE layer. Is that okay to your expectation?
Pytorch's NCCL operations (e.g. AllToAll) don't support pickle, so I'm afraid that any MoE models having AllToAll in their forward pass will have the same issue. You may either ask Torch community to fix it up, or you do any workaround in ema that doesn't require deepcopy, or you can only boot MoE model in data parallel mode, which wouldn't have AllToAll in the forward pass, though the distributed performance will be very bad in large scale.
Thanks for your prompt reply! I use 8 GPUs to train my model with 8 experts. In that case, can I pickle my model for a single GPU? I also would like to know if it is a requirement to save the checkpoints separately for each rank? Does it always need 8 GPUs for inference if I trained the model with 8 GPUs?
You can go through these examples to convert training checkpoints between distributed version and single-device version: https://github.com/microsoft/tutel#how-to-convert-checkpoint-files-that-adapt-to-different-distributed-world-sizes
Thanks for your quick update for this feature! I notice you use mpiexec to launch the job and save the ckpt. If I use torch.distributed.launch to train my moe, is it still valid to use the tutel/checkpoint/gather.py to combine my checkpoints?
Thanks for your quick update for this feature! I notice you use mpiexec to launch the job and save the ckpt. If I use torch.distributed.launch to train my moe, is it still valid to use the tutel/checkpoint/gather.py to combine my checkpoints?
Yes, both are compatible, as mpiexec
is just an alternative way to launch cross-node processes instead of torch.distributed.launch
.
Hi, thanks for this awesome project!
I build my transformer model based on the MoeMlp layer. I use ema for better performance. However, when I trying to init my ema model with
ema_model = copy.deepcopy(my_transformer_model)
, I encounter the error:Could you help me with that? How can I use ema with tutel? Thanks!