DistributedDataParallel device_ids and output_device arguments only work with single-device CUDA modules

[tengerye]

Hi, I encountered a problem with your code:

AssertionError: DistributedDataParallel device_ids and output_device arguments only work
with single-device CUDA modules, but got device_ids [0], output_device 0, and
module parameters {device(type='cuda', index=0), device(type='cpu')}.

But if I commented

if self.config['trainer']['sync_batch_norm']:
    self.model = torch.nn.SyncBatchNorm.convert_sync_batchnorm(self.model)

of your trainer.py, the code runs without errors. I used the recommended settings. May I ask why?

[wenwenyu]

SyncBatchNorm is only working under the condition of one GPU per-process mode according to requirements of official Pytorch. In other words, the nproc_per_node and local_world_size args must be the same in train command args.

So the reasons for the error are that you turn on sync_batch_norm arg as true in config.json but the nproc_per_node and local_world_size args is different when you run commends. Please check again.

[tengerye]

Hi @wenwenyu , thanks a lot for your kind reply. I did set the number of nproc_per_node and local_world_size to be the same. I set both of them to 1.

[wenwenyu]

Try to set the number of CUDA_VISIBLE_DEVICES as same as nproc_per_node.

[wenwenyu]

I am going to close this issue. Please feel free to reopen or create a new one if you have more questions.