search

skyhehe123 / SA-SSD

SA-SSD: Structure Aware Single-stage 3D Object Detection from Point Cloud (CVPR 2020)
492 stars 106 forks source link

cuda execution failed with error 35 #60

Open songw-zju opened 4 years ago

songw-zju commented 4 years ago

when I run train.py, there are lots of errors in numba, and an error occrured: Traceback (most recent call last): File "tools/train.py", line 130, in main() File "tools/train.py", line 120, in main log_interval = cfg.log_config.interval File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 95, in train_model log_interval = log_interval File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 57, in train_one_epoch outputs = batch_processor(model, data_batch) File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 29, in batch_processor losses = model(data) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(inputs[0], kwargs[0]) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/detectors/base.py", line 79, in forward return self.forward_train(img, img_meta, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/detectors/single_stage.py", line 83, in forward_train (x, conv6), point_misc = self.neck(vx, ret['coordinates'], batch_size) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/necks/cmn.py", line 107, in forward x, point_misc = self.backbone(x, points_mean, is_test) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/necks/cmn.py", line 203, in forward x = self.conv0(x) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward input = module(input) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/conv.py", line 151, in forward self.stride, self.padding, self.dilation, self.output_padding, self.subm, self.transposed, grid=input.grid) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/ops.py", line 89, in get_indice_pairs stride, padding, dilation, out_padding, int(subm), int(transpose)) RuntimeError: /home/weijk/SA-SSD/spconv_8da6f96/src/spconv/indice.cu 120 cuda execution failed with error 35

AOOOOOA commented 1 year ago

when I run train.py, there are lots of errors in numba, and an error occrured: Traceback (most recent call last): File "tools/train.py", line 130, in main() File "tools/train.py", line 120, in main log_interval = cfg.log_config.interval File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 95, in train_model log_interval = log_interval File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 57, in train_one_epoch outputs = batch_processor(model, data_batch) File "/home/weijk/ws/SA-SSD/tools/train_utils/init.py", line 29, in batch_processor losses = model(data) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, *kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward return self.module(inputs[0], kwargs[0]) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/detectors/base.py", line 79, in forward return self.forward_train(img, img_meta, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/detectors/single_stage.py", line 83, in forward_train (x, conv6), point_misc = self.neck(vx, ret['coordinates'], batch_size) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/necks/cmn.py", line 107, in forward x, point_misc = self.backbone(x, points_mean, is_test) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, kwargs) File "/home/weijk/ws/SA-SSD/mmdet/models/necks/cmn.py", line 203, in forward x = self.conv0(x) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call* result = self.forward(input, kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/modules.py", line 123, in forward input = module(input) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/conv.py", line 151, in forward self.stride, self.padding, self.dilation, self.output_padding, self.subm, self.transposed, grid=input.grid) File "/home/weijk/anaconda3/envs/SASSD/lib/python3.6/site-packages/spconv/ops.py", line 89, in get_indice_pairs stride, padding, dilation, out_padding, int(subm), int(transpose)) RuntimeError: /home/weijk/SA-SSD/spconv_8da6f96/src/spconv/indice.cu 120 cuda execution failed with error 35

Hi I also met this problem. Could you please tell me how did you solve this problem?