Open Gumpest opened 2 years ago
You can use PyTorch Lightning instead. It automatically parallelizes the model training across GPUs and also supports TPU with just a single argument.
We use the nccl backend with PyTorch to parallelize the stream while inference (testing). For training, we use usual distributed setup.
So we need at least three GPUs to inference three streams in ParNet?
Yes if you want to do the multi-GPU inference. Otherwise, you can also do single gpu inference but it will be slower.
@imankgoyal
foe the edge devcie, using mulit-gpu for inference is expensive, what is your opinion?
Could you introduce more details in parallelizing across GPUs, like how to implement through PyTorch.