search

loicland / superpoint_graph

Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs
MIT License
758 stars 214 forks source link

cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered #285

Closed whuhxb closed 1 year ago

whuhxb commented 1 year ago

Hi @loicland @nicolas-chaulet @mys007 @bermanmaxim @Stakhan

Have you ever met this bug before? Thanks.

Traceback (most recent call last): File "learning/main.py", line 395, in main() File "learning/main.py", line 291, in main acc, loss, oacc, avg_iou = train() File "learning/main.py", line 188, in train outputs = model.ecc(embeddings) File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(*input, *kwargs) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/graphnet.py", line 97, in forward input = module(input) File "/export/home/hanxiaobing/anaconda3/envs/SPG/lib/python3.7/site-packages/torch/nn/modules/module.py", line 477, in call result = self.forward(input, **kwargs) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/modules.py", line 54, in forward input = ecc.GraphConvFunction(nc, nc, idxn, idxe, degs, degs_gpu, self._edge_mem_limit)(hx, weights) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/GraphConvModule.py", line 67, in forward cuda_kernels.conv_aggregate_fw(output.narrow(0,startd,numd), products.view(-1,self._out_channels), self._degs_gpu.narrow(0,startd,numd)) File "/export/home/hanxiaobing/Documents/PlaneNet_PlaneRCNN/DGCNN_PointNet2/SensatUrban/SPG/superpoint_graph-release/learning/ecc/cuda_kernels.py", line 124, in conv_aggregate_fw block=(CUDA_NUM_THREADS,1,1), grid=(GET_BLOCKS(w),n//blockDimY+1,1), stream=stream) File "cupy/cuda/function.pyx", line 182, in cupy.cuda.function.Function.call File "cupy/cuda/function.pyx", line 164, in cupy.cuda.function._launch File "cupy_backends/cuda/api/driver.pyx", line 299, in cupy_backends.cuda.api.driver.launchKernel File "cupy_backends/cuda/api/driver.pyx", line 124, in cupy_backends.cuda.api.driver.check_status cupy_backends.cuda.api.driver.CUDADriverError: CUDA_ERROR_ILLEGAL_ADDRESS: an illegal memory access was encountered

loicland commented 1 year ago

which version of PyTorch /CUDA are you using?

whuhxb commented 1 year ago

which version of PyTorch /CUDA are you using?

pytorch 0.4.0 cuda11.1, and now downgrade cuda11.1 to cuda90, and trying. Which version of pytorch and cuda should I use? Thanks.

loicland commented 1 year ago

Hi!

We are releasing a new version of SuperPoint Graph called SuperPoint Transformer (SPT).

https://github.com/drprojects/superpoint_transformer

It is better in any way:

✨ SPT in numbers ✨
📊 SOTA results: 76.0 mIoU S3DIS 6-Fold, 63.5 mIoU on KITTI-360 Val, 79.6 mIoU on DALES
🦋 212k parameters only!
⚡ Trains on S3DIS in 3h on 1 GPU
⚡ Preprocessing is x7 faster than SPG!
🚀 Easy install (no more boost!)

If you are interested in lightweight, high-performance 3D deep learning, you should check it out. In the meantime, we will finally retire SPG and stop maintaining this repo.