Open anqin5211314 opened 4 years ago
This error tells that some of the class values in the pixels of your masks are greater than n_classes. Try this:
img= cv2.imread(path_to_mask)
print(numpy.unique(img, return_counts= True))
This will show the unique pixel values and their counts. Check if you have any pixel values more than n_classes.
/pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , long , T , int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [987,0,0] Assertion
trainer.train()
File "train.py", line 235, in train
losses.backward()
File "/home/anqin/anaconda3/envs/ICnet/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
torch.autograd.backward(self, gradient, retain_graph, create_graph)
File "/home/anqin/anaconda3/envs/ICnet/lib/python3.6/site-packages/torch/autograd/init.py", line 93, in backward
allow_unreachable=True) # allow_unreachable flag
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
t >= 0 && t < n_classes
failed. /pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , long , T , int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [988,0,0] Assertiont >= 0 && t < n_classes
failed. /pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , long , T , int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [989,0,0] Assertiont >= 0 && t < n_classes
failed. /pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , long , T , int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [990,0,0] Assertiont >= 0 && t < n_classes
failed. /pytorch/aten/src/THCUNN/SpatialClassNLLCriterion.cu:103: void cunn_SpatialClassNLLCriterion_updateOutput_kernel(T , T , T , long , T *, int, int, int, int, int, long) [with T = float, AccumT = float]: block: [0,0,0], thread: [991,0,0] Assertiont >= 0 && t < n_classes
failed. Traceback (most recent call last): File "train.py", line 331, in