search

zijundeng / pytorch-semantic-segmentation

PyTorch for Semantic Segmentation
MIT License
1.73k stars 393 forks source link

cuDNN error: CUDNN_STATUS_NOT_INITIALIZED #61

Open AnhNguyenUK opened 4 years ago

AnhNguyenUK commented 4 years ago

I training FCN32 with the my onw datatset. However, i realized that the loss is empty and lead to the error above. The detailed error: C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [293,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [294,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [32,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [33,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [34,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [35,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [514,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [515,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [516,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [517,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [518,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [160,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [161,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [162,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [163,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [164,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [165,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [166,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [167,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [168,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [256,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [257,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [258,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [259,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [260,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [261,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [262,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [384,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [385,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [386,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [387,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [388,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [389,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [390,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [391,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [392,0,0] Assertion t >= 0 && t < n_classes failed. C:/cb/pytorch_1000000000000/work/aten/src/THCUNN/SpatialClassNLLCriterion.cu:106: block: [3,0,0], thread: [393,0,0] Assertion t >= 0 && t < n_classes failed. Traceback (most recent call last): File "train.py", line 245, in main(args) File "train.py", line 126, in main train(train_loader, net, criterion, optimizer, epoch, train_args) File "train.py", line 149, in train loss.backward() File "C:\Users\ADMIN\anaconda3\envs\Pytorch\lib\site-packages\torch\tensor.py", line 198, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "C:\Users\ADMIN\anaconda3\envs\Pytorch\lib\site-packages\torch\autograd__init__.py", line 100, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED (createCuDNNHandle at ..\aten\src\ATen\cudnn\Handle.cpp:9) (no backtrace available).