Closed BrukArkady closed 2 years ago
Check the device type of the tensors at the error line. Move them all to either GPU or cpu and run.
Check the device type of the tensors at the error line. Move them all to either GPU or cpu and run.
@p0p4k Can you elaborate a bit, please? I'm having the same problem.
@BrukArkady The answer is given by @p0p4k. To be precise, you should change line #3150
of functional.py
from:
return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
to:
return torch.binary_cross_entropy_with_logits(input.cuda(), target.cuda(), weight, pos_weight.cuda(), reduction_enum)
For precise analysis of the error, append the following code, you can see the tensors' respective devices. In the TTS/TTS/tts/layers/losses.py
file,
#add immediately above L193
tensors_to_check = [x.masked_select(mask), target.masked_select(mask), self.pos_weight]
for t in tensors_to_check:
try:
print(f'tensor {t} is on GPU device - {t.get_device()}
except:
print(f'tensor {t} is on cpu'}
#add immediately above L197
tensors_to_check = [x, target, pos_weight=self.pos_weight]
for t in tensors_to_check:
try:
print(f'tensor {t} is on GPU device - {t.get_device()}
except:
print(f'tensor {t} is on cpu'}
Then you can just add tensor.cuda()
to change a tensor's device to GPU. You can do this directly without doing the above step as well.
@BrukArkady The answer is given by @p0p4k. To be precise, you should change
line #3150
offunctional.py
from:return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
to:
return torch.binary_cross_entropy_with_logits(input.cuda(), target.cuda(), weight, pos_weight.cuda(), reduction_enum)
Would prefer doing the changes in TTS file, rather than Pytorch library.
@BrukArkady The answer is given by @p0p4k. To be precise, you should change
line #3150
offunctional.py
from:return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
to:
return torch.binary_cross_entropy_with_logits(input.cuda(), target.cuda(), weight, pos_weight.cuda(), reduction_enum)
Would prefer doing the changes in TTS file, rather than Pytorch library.
Got it. Thanks.
@BrukArkady The answer is given by @p0p4k. To be precise, you should change
line #3150
offunctional.py
from:return torch.binary_cross_entropy_with_logits(input, target, weight, pos_weight, reduction_enum)
to:
return torch.binary_cross_entropy_with_logits(input.cuda(), target.cuda(), weight, pos_weight.cuda(), reduction_enum)
Would prefer doing the changes in TTS file, rather than Pytorch library.
Got it. Thanks.
Then again one issue is if you are using a machine with no GPU, we have to write if-else for that. Can you make a PR?
Then again one issue is if you are using a machine with no GPU, we have to write if-else for that. Can you make a PR?
I'm afraid I can't right now. I'll look into it once I get some spare time.
same exact issue here... can someone comment on this issue when it is fixed, so i can try again... Thanks
Fixed by #1872
Describe the bug
I'm trying to run Tacotron2 training, but receives
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
To Reproduce
CUDA_VISIBLE_DEVICES="0" python3 train_tacotron_ddc.py
Expected behavior
No response
Logs
Environment
Additional context
No response