Question about cross-node(multi-node) data parallelism on GPU

google-research / text-to-text-transfer-transformer

Code for the paper "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer"

https://arxiv.org/abs/1910.10683

Apache License 2.0

6.18k stars 757 forks source link

Closed hwyFighting closed 1 year ago

hwyFighting commented 1 year ago

Hi! I would like to ask how to get t5 to train data in parallel across nodes on GPUs?

Is there a corresponding solution in the case of mesh_tensorflow based? Or there is no corresponding design yet?

Thank you for your positive and exact reply!

adarob commented 1 year ago