hi, i am running train_depth with multi gpus, and find a problem that each gpu processes the same subset from the dataloader. I am using the default multi-gpu mode, ie strategy=‘ddp’ . However, by printing the loaded data of training batches, I found the inputs to all gpu nodes are the same. I think the problem lies in that the original WeightedRandomSampler does not apply to DistributedDataParallel.
hi, i am running train_depth with multi gpus, and find a problem that each gpu processes the same subset from the dataloader. I am using the default multi-gpu mode, ie strategy=‘ddp’ . However, by printing the loaded data of training batches, I found the inputs to all gpu nodes are the same. I think the problem lies in that the original WeightedRandomSampler does not apply to DistributedDataParallel.