search

meidachen / STPLS3D

🔥 Synthetic and real-world 2d/3d dataset for semantic and instance segmentation (BMVC 2022 Oral)
236 stars 20 forks source link

CUDA kernel failed : an illegal memory access was encountered #9

Open yanx27 opened 2 years ago

yanx27 commented 2 years ago

Hi @meidachen , I am currently run the HAIS model for instance segmentation. After the training process, I meet an error when running python test.py --config config/hais_run_stpls3d.yaml --pretrain exp/Synthetic_v3_InstanceSegmentation/hais/hais_run_stpls3d/hais_run_stpls3d-000000500.pth :

CUDA kernel failed : an illegal memory access was encountered

I traced the problem and found the code breaks around the following lines: https://github.com/meidachen/STPLS3D/blob/8b83424494fd3d1063b4d40d104465b3d6ceb4ae/HAIS/model/hais/hais.py#L306

Could you have some suggestions? Looking forwarding to your reply!

meidachen commented 2 years ago

Hi @yanx27 , it seems like you are having the same issue as https://github.com/hustvl/HAIS/issues/21#issue-1152840222. The authors suggested to: "It seems a problem of memory overflow. If point clouds of your datasets are much denser or larger than ScanNet, I suggest to increase the values of these macros."

https://github.com/meidachen/STPLS3D/blob/8b83424494fd3d1063b4d40d104465b3d6ceb4ae/HAIS/lib/hais_ops/src/hierarchical_aggregation/hierarchical_aggregation.cu#L8toL11

It maybe because you are using a GPU which has less RAM than I have (3090), so changing these parameters may work.