Multi GPU support - Githubissues

google-research / albert

ALBERT: A Lite BERT for Self-supervised Learning of Language Representations

Apache License 2.0

3.25k stars 570 forks source link

Multi GPU support #68

Closed matteodelv closed 4 years ago

matteodelv commented 4 years ago

Is multi-GPU support expected to be implemented anytime soon? What are the conceptual changes that need to be done in order to make ALBERT train/fine-tune on multiple GPUs? Should TPUEstimator be replaced by Estimator, together with a distributed strategy such as MirroredStrategy? Thank you so much!

0x0539 commented 4 years ago

Unfortunately, I don't think we'll be able to get to this anytime soon. Maybe one of the other ALBERT implementations will have it?

apohllo commented 4 years ago

https://github.com/kamalkraj/ALBERT-TF2.0 has support for multi-GPU pre-training