search

nv-tlabs / GSCNN

Gated-Shape CNN for Semantic Segmentation (ICCV 2019)
https://nv-tlabs.github.io/GSCNN/
Other
916 stars 200 forks source link

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function #90

Open nrose171 opened 1 year ago

nrose171 commented 1 year ago

Hello, I am wondering if anyone else has gotten this THCudaCheck Fail error while running this. I am running cuda 12.2v with an NVIDIA RTX A3000 Laptop GPU. Please let me know if you figure anything out.

Here is my full output after running:

07-19 23:21:43.075 train fine cities: ['train/aachen', 'train/bochum', 'train/bremen', 'train/cologne', 'train/darmstadt', 'train/dusseldorf', 'train/erfurt', 'train/hamburg', 'train/hanover', 'train/jena', 'train/krefeld', 'train/monchengladbach', 'train/strasbourg', 'train/stuttgart', 'train/tubingen', 'train/ulm', 'train/weimar', 'train/zurich'] 07-19 23:21:43.083 Cityscapes-train: 2975 images 07-19 23:21:43.083 val fine cities: ['val/frankfurt', 'val/lindau', 'val/munster'] 07-19 23:21:43.085 Cityscapes-val: 500 images 07-19 23:21:43.085 Using Per Image based weighted loss /home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/loss.py:216: UserWarning: NLLLoss2d has been deprecated. Please use NLLLoss instead as a drop-in replacement and see https://pytorch.org/docs/master/nn.html#torch.nn.NLLLoss for more details. warnings.warn("NLLLoss2d has been deprecated. " /home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/_reduction.py:46: UserWarning: size_average and reduce args will be deprecated, please use reduction='mean' instead. warnings.warn(warning.format(ret)) 07-19 23:21:43.085 Using Cross Entropy Loss Creating wide resnet /home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/encoding/nn/syncbn.py:149: EncodingDeprecationWarning: encoding.nn.BatchNorm2d is now deprecated in favor of encoding.nn.SyncBatchNorm. .format('BatchNorm2d', SyncBatchNorm.name), EncodingDeprecationWarning) /home/nrose171/Documents/GitHub/GSCNN/network/mynn.py:29: UserWarning: nn.init.kaiming_normal is now deprecated in favor of nn.init.kaimingnormal. nn.init.kaiming_normal(module.weight) 07-19 23:21:44.878 Model params = 32.2M 07-19 23:21:44.895 Loading weights from model checkpoints/best_cityscapes_checkpoint.pth 07-19 23:21:45.211 Load Compelete

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=383 error=8 : invalid device function Traceback (most recent call last): File "train.py", line 383, in main() File "train.py", line 142, in main optim, default_eval_epoch, writer) File "train.py", line 302, in validate seg_out, edge_out = net(input) # output = (1, 19, 713, 713) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(*inputs[0], *kwargs[0]) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/nrose171/Documents/GitHub/GSCNN/network/gscnn.py", line 272, in forward m1 = self.mod1(inp) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, **kwargs) File "/home/nrose171/anaconda3/envs/GSCNN_37/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 338, in forward self.padding, self.dilation, self.groups) RuntimeError: cuda runtime error (8) : invalid device function at /pytorch/aten/src/THC/THCGeneral.cpp:383