How to run microbatches on different gpus?

tensorflow / lingvo

Lingvo

Apache License 2.0

2.81k stars 441 forks source link

How to run microbatches on different gpus? #256

Open adis98 opened 3 years ago

adis98 commented 3 years ago

Hi,

I've been experimenting with GPipe and was wondering if it is possible to run different micro-batches on different GPUS? For example if there are 16 micro-batches, is it possible to run 8 micro-batches on GPU 0 and the other 8 on GPU 1? It would be faster than pipelining all the micro-batches on the same machine right?

adis98 commented 3 years ago

@jonathanasdf, could you help me with this?

jonathanasdf commented 3 years ago

Sorry I'm not familiar with GPipe. @bignamehyp ?

adis98 commented 3 years ago

@bignamehyp , if the above mentioned scenario is not possible....does lingvo support data-parallelism? i.e, running multiple minibatches in parallel and then aggregating their results? If yes, what are the command line arguments for it?