search

gpgpu-sim / pytorch-gpgpu-sim

Modified version of PyTorch able to work with changes to GPGPU-Sim
Other
46 stars 25 forks source link

Error with RuntimeError: CuDNN error: CUDNN_STATUS_MAPPING_ERROR #11

Open superQCman opened 4 months ago

superQCman commented 4 months ago 
Traceback (most recent call last):
  File "main.py", line 155, in <module>
    main()
  File "main.py", line 142, in main
    train(args, model, device, train_loader, optimizer, epoch)
  File "main.py", line 44, in train
    output = model(data)
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "main.py", line 24, in forward
    x = self.conv2(x)
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CuDNN error: CUDNN_STATUS_MAPPING_ERROR

How to solve this problem? I use cuda 10.1, cudnn 7.6.5, python 3.6

superQCman commented 4 months ago 
Traceback (most recent call last):
  File "main.py", line 155, in <module>
    torch.cuda.set_device(0)
  File "main.py", line 142, in main
    for epoch in range(1, args.epochs + 1):
  File "main.py", line 44, in train
    optimizer.zero_grad()
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "main.py", line 24, in forward
    x=x.float()
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/sim/anaconda3/envs/gpgpu-sim/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: cublas runtime error : the GPU program failed to execute at /home/sim/pytorch-gpgpusim/aten/src/THC/THCBlas.cu:249
makefile:7: recipe for target 'run-main' failed
make: *** [run-main] Error 1

After I disabled cudnn, the Error changed to this

itsMaoMao commented 1 month ago

i found this, https://github.com/gpgpu-sim/pytorch-gpgpu-sim/issues/5