sacmehta / ESPNetv2

A light-weight, power efficient, and general purpose convolutional neural network
MIT License
446 stars 70 forks source link

RuntimeError: cuda runtime error (59) : device-side assert triggered at /pytorch/aten/src/THC/generic/THCTensorCopy.cpp:70 #4

Closed monk42 closed 5 years ago

monk42 commented 5 years ago

When I run the segmentation codes , i got this error!
59fdee984bae45d85735b2e52b0e8b5

It seems that there is a problem in calculating the loss function!The loss value cannot be calculated.

I run this codes in pytorch0.4.1 and python3 and opencv 3.2.0

sacmehta commented 5 years ago

Does your machine has GPU with CUDa?

monk42 commented 5 years ago

yes, it's cuda 8.0, cudnn5.1 and gpu is 1050ti!

sacmehta commented 5 years ago

You might want to upgrade it and try. I was using Cuda 9.0 and cudnn 7

monk42 commented 5 years ago

OK I will try i, thanks!

monk42 commented 5 years ago

Thanks, I have solved this problem. It turned out that I didn't convert 255 of the input label to 19, not because of the problem of the CUDA version. There are 255 label in the voc12 data set. How can I handle this data set to use your network properly?

sacmehta commented 5 years ago

You can ignore it in the loss function.

nn.CrossEntropy(ignore_index=255)

monk42 commented 5 years ago

If I set nClasses = 21,is 255 label also ignored in IOUEval.py ?

sacmehta commented 5 years ago

The easiest way is to do something like this:

https://github.com/sacmehta/ESPNetv2/blob/df58a62b32ae17074ab7a8610bd319365f7eb85f/segmentation/DataSet.py#L40

For cityscapes, I map 255 to 19. For VOC, you can map 255 to 0

monk42 commented 5 years ago

I really appreciate your answer, it's help me a lot. I'll try the approach you suggested. Thanks !

monk42 commented 5 years ago
When you test ESPNetv2 on camvid dataset, did you augment the dataset and did you ignore the background classes when training ?
sacmehta commented 5 years ago

We didn’t train on CamVid dataset.

monk42 commented 5 years ago

sorry,I was mistaken.