Closed tengerye closed 4 years ago
SyncBatchNorm
is only working under the condition of one GPU per-process mode according to requirements of official Pytorch. In other words, the nproc_per_node
and local_world_size
args must be the same in train command args.
So the reasons for the error are that you turn on sync_batch_norm
arg as true
in config.json
but the nproc_per_node
and local_world_size
args is different when you run commends. Please check again.
Hi @wenwenyu , thanks a lot for your kind reply. I did set the number of nproc_per_node
and local_world_size
to be the same. I set both of them to 1.
Try to set the number of CUDA_VISIBLE_DEVICES
as same as nproc_per_node
.
I am going to close this issue. Please feel free to reopen or create a new one if you have more questions.
Hi, I encountered a problem with your code:
But if I commented
of your
trainer.py
, the code runs without errors. I used the recommended settings. May I ask why?