dearleiii
commented
6 years ago Model is DataParallel model:
leichen@gpu-compute3$ python3 load_model_test.py DataParallel( (module): APXM_conv3( (main): Sequential( (0): Conv2d(3, 8, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)) (1): LeakyReLU(negative_slope=0.2, inplace) (2): Conv2d(8, 16, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1)) (3): LeakyReLU(negative_slope=0.2, inplace) (4): Conv2d(16, 32, kernel_size=(4, 4), stride=(2, 2), padding=(1, 1), bias=False) (5): LeakyReLU(negative_slope=0.2, inplace) ) (regressor): Sequential( (0): Linear(in_features=1048576, out_features=256, bias=True) (1): LeakyReLU(negative_slope=0.01) (2): Linear(in_features=256, out_features=1, bias=True) ) )
(regressor): Sequential( (0): Linear(in_features=1048576, out_features=256, bias=True) (1): LeakyReLU(negative_slope=0.01) (2): Linear(in_features=256, out_features=1, bias=True) ) ) ) intpus: torch.Size([50, 3, 1020, 2040]) scores: torch.Size([50, 1]) THCudaCheck FAIL file=torch/csrc/cuda/Module.cpp line=32 error=10 : invalid device ordinal Traceback (most recent call last): File "load_model_test.py", line 69, in
outputs = model1(inputs)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/modules/module.py", line 491, in call
result = self.forward(*input, *kwargs)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 110, in forward
inputs, kwargs = self.scatter(inputs, kwargs, self.device_ids)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/data_parallel.py", line 121, in scatter
return scatter_kwargs(inputs, kwargs, device_ids, dim=self.dim)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 36, in scatter_kwargs
inputs = scatter(inputs, target_gpus, dim) if inputs else []
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 29, in scatter
return scatter_map(inputs)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 16, in scatter_map
return list(zip(map(scatter_map, obj)))
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/scatter_gather.py", line 14, in scatter_map
return Scatter.apply(target_gpus, None, dim, obj)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/nn/parallel/_functions.py", line 74, in forward
outputs = comm.scatter(input, ctx.target_gpus, ctx.chunk_sizes, ctx.dim, streams)
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/comm.py", line 157, in scatter
with torch.cuda.device(device), torch.cuda.stream(stream):
File "/home/home2/leichen/.local/lib/python3.5/site-packages/torch/cuda/init.py", line 227, in enter
torch._C._cuda_setDevice(self.idx)
RuntimeError: cuda runtime error (10) : invalid device ordinal at torch/csrc/cuda/Module.cpp:32