search

google-research / electra

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators
Apache License 2.0
2.33k stars 352 forks source link

pretrain with multigpu #107

Open 652994331 opened 3 years ago

652994331 commented 3 years ago

Hi, guys, can we use multi-gpus to pretrain electra-small or electra-base now. Anyone figured it out? thanks

hiroshi-matsuda-rit commented 3 years ago

Check out the NVIDIA's implementation. https://github.com/NVIDIA/DeepLearningExamples/tree/master/TensorFlow2/LanguageModeling/ELECTRA

PhilipMay commented 3 years ago

Or this: https://github.com/richarddwang/electra_pytorch

MrSworder commented 2 years ago

Or this: https://github.com/richarddwang/electra_pytorch

Hello，I tried electra_pytorch, but only one GPU was running. Have you ever run it with multigpu? How did you do that?