Closed stas00 closed 3 years ago
Pinging @JetRunner
@stas00 Well it is just not designed for DP or DDP. DeeBERT is for accelerating inference with bs=1 (especially on CPU). I don't believe it should support DP.
But yes theoretically it can support multi-GPU training but I'm not sure if it's necessary?
That's good enough for me, I will leave it at 0 or 1-gpu - no problem - thank you for elaborating about the needs of this example, @JetRunner!
I'm working on making the tests work under multiple gpus and run into and this one that proved to be stubborn, for some reason it doesn't work under any DP scheme. I don't know anything about this script, To reproduce:
Note - you need at least 2 gpus:
Actually it fails with 1 gpu too (just change to --nproc_per_node=1)
@LysandreJik