Error Cuda

[youyeg]

Hi, this is the error I got while executing this comman, could you please check this?

python3 train.py --model_dir experiments/resnet18_distill/resnext_teacher Loading the datasets... Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified

• done. /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:82: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.classifier.weight) /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:87: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.state_dict()[key], mode='fan_out') THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error Traceback (most recent call last): File "train.py", line 421, in teacher_model = nn.DataParallel(teacher_model).cuda() File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/sitepython3 train.py --model_dir experiments/resnet18_distill/resnext_teacher Loading the datasets... Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified
• done. /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:82: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.classifier.weight) /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:87: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.state_dict()[key], mode='fan_out') THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error Traceback (most recent call last): File "train.py", line 421, in teacher_model = nn.DataParallel(teacher_model).cuda() File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply param.data = fn(param.data) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51 -packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply param.data = fn(param.data) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51

[haitongli]

Hi, this is the error I got while executing this comman, could you please check this?

python3 train.py --model_dir experiments/resnet18_distill/resnext_teacher Loading the datasets... Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified

• done. /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:82: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.classifier.weight) /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:87: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.state_dict()[key], mode='fan_out') THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error Traceback (most recent call last): File "train.py", line 421, in teacher_model = nn.DataParallel(teacher_model).cuda() File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/sitepython3 train.py --model_dir experiments/resnet18_distill/resnext_teacher Loading the datasets... Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified Files already downloaded and verified
• done. /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:82: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.classifier.weight) /u/halle/yeganeh/home_at/Desktop/git/knowledge-distillation-pytorch/model/resnext.py:87: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. init.kaiming_normal(self.state_dict()[key], mode='fan_out') THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=51 error=30 : unknown error Traceback (most recent call last): File "train.py", line 421, in teacher_model = nn.DataParallel(teacher_model).cuda() File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in cuda return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply param.data = fn(param.data) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51 -packages/torch/nn/modules/module.py", line 187, in _apply module._apply(fn) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 193, in _apply param.data = fn(param.data) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/nn/modules/module.py", line 260, in return self._apply(lambda t: t.cuda(device)) File "/u/halle/yeganeh/home_at/.local/lib/python3.6/site-packages/torch/cuda/init.py", line 162, in _lazy_init torch._C._cuda_init() RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCGeneral.cpp:51

From "* RuntimeError: cuda runtime error (30)", it looks like the error has to do with your GPU and CUDA environment. Can you confirm that that you have GPU and CUDA driver configured correctly?