binary_op(): expected both inputs to be on same device

[JoyHuYY1412]

Hi, sorry to interrupt you. I use your code which with 4 gpus, but I use the 4,5,6,7 gpu on my machine. However, when I try to resume my model, there is an error says:

Traceback (most recent call last):
  File "upsnet/upsnet_end2end_train.py", line 418, in <module>
    upsnet_train()
  File "upsnet/upsnet_end2end_train.py", line 300, in upsnet_train
    optimizer.step(lr)
  File "upsnet/../lib/nn/optimizer.py", line 98, in step
    buf.mul_(momentum).add_(group['lr'] * lr, d_p)
RuntimeError: binary_op(): expected both inputs to be on same device, but input a is on cuda:0 and input b is on cuda:4

Can you tell me what to do? I really appreciate your help.

[YuwenXiong]

If you use 4,5,6,7 GPU, the best way is to set CUDA_VISIBLE_DEVICES=4,5,6,7 and keep using 0,1,2,3 in the config files