search

stevenygd / PointFlow

PointFlow : 3D Point Cloud Generation with Continuous Normalizing Flows
https://www.guandaoyang.com/PointFlow/
MIT License
720 stars 101 forks source link

CUDA kernel error #7

Closed cookiew closed 4 years ago

cookiew commented 4 years ago

Hi, thank you for sharing the code!

I was trying to run the evaluation code with CUDA_VISIBLE_DEVICES=0 ./scripts/shapenet_airplane_ae_test.sh but got the following error:

  File "test.py", line 167, in <module>
    main(args)
  File "test.py", line 159, in main
    evaluate_recon(model, args)
  File "test.py", line 69, in evaluate_recon
    results = EMD_CD(sample_pcs, ref_pcs, args.batch_size, accelerated_cd=True)
  File "/home/xxx/PointFlow/metrics/evaluation_metrics.py", line 66, in EMD_CD
    cd_lst.append(dl.mean(dim=1) + dr.mean(dim=1))
RuntimeError: CUDA error: no kernel image is available for execution on the device

I am using Tesla K80 GPU. And I installed dependencies as instructed with python 3.6.9, using the install.sh file ( CUDA 10.0, GCC 5.2.0, pytorch 1.0.1). install.sh gives no error (see below for part of the output I think is relevant). 

CXX src/structural_loss.cpp
g++ src/structural_loss.cpp -MMD -MP -pthread -fPIC -fwrapv -std=c++11 -DDEBUG -g -O0 -I./ -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -DTORCH_API_INCLUDE_EXTENSION_H -D_GLIBCXX_USE_CXX11_ABI=0 -Wall -Wno-sign-compare -Wcomment -c -o objs/structural_loss.o
NVCC src/approxmatch.cu
/usr/local/cuda/bin/nvcc -g -G  -std=c++11 -ccbin=g++ -Xcompiler -fPIC -DDEBUG -g -O0 -I./ -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -DTORCH_API_INCLUDE_EXTENSION_H -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_52,code=sm_52 -M src/approxmatch.cu -o objs/cuda/approxmatch.d \
    -odir objs/cuda
/usr/local/cuda/bin/nvcc -g -G  -std=c++11 -ccbin=g++ -Xcompiler -fPIC -DDEBUG -g -O0 -I./ -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -DTORCH_API_INCLUDE_EXTENSION_H -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_52,code=sm_52 -c src/approxmatch.cu -o objs/cuda/approxmatch.o
NVCC src/nndistance.cu
/usr/local/cuda/bin/nvcc -g -G  -std=c++11 -ccbin=g++ -Xcompiler -fPIC -DDEBUG -g -O0 -I./ -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -DTORCH_API_INCLUDE_EXTENSION_H -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_52,code=sm_52 -M src/nndistance.cu -o objs/cuda/nndistance.d \
    -odir objs/cuda
/usr/local/cuda/bin/nvcc -g -G  -std=c++11 -ccbin=g++ -Xcompiler -fPIC -DDEBUG -g -O0 -I./ -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -DTORCH_API_INCLUDE_EXTENSION_H -D_GLIBCXX_USE_CXX11_ABI=0 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61 -gencode arch=compute_52,code=sm_52 -c src/nndistance.cu -o objs/cuda/nndistance.o
rm -f -f ./objs/libmake_pytorch.a
rm -f -rf build dist
LD -o objs/libmake_pytorch.a
ar rc ./objs/libmake_pytorch.a ./objs/structural_loss.o ./objs/cuda/approxmatch.o ./objs/cuda/nndistance.o
python setup.py build
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/StructuralLosses
copying ./__init__.py -> build/lib.linux-x86_64-3.6/StructuralLosses
copying ./nn_distance.py -> build/lib.linux-x86_64-3.6/StructuralLosses
copying ./match_cost.py -> build/lib.linux-x86_64-3.6/StructuralLosses
running build_ext
building 'StructuralLossesBackend' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/pybind
gcc -pthread -B /home/xxx/anaconda3/envs/PointFlow/compiler_compat -Wl,--sysroot=/ -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I./ -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/torch/csrc/api/include -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/TH -I/home/xxx/anaconda3/envs/PointFlow/lib/python3.6/site-packages/torch/lib/include/THC -I/usr/local/cuda/include -I/home/xxx/anaconda3/envs/PointFlow/include/python3.6m -c pybind/bind.cpp -o build/temp.linux-x86_64-3.6/pybind/bind.o -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=StructuralLossesBackend -D_GLIBCXX_USE_CXX11_ABI=0 -std=c++11
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
g++ -pthread -shared -B /home/xxx/anaconda3/envs/PointFlow/compiler_compat -L/home/xxx/anaconda3/envs/PointFlow/lib -Wl,-rpath=/home/xxx/anaconda3/envs/PointFlow/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/pybind/bind.o -Lobjs -L/usr/local/cuda/lib64 -lmake_pytorch -lcudart -o build/lib.linux-x86_64-3.6/StructuralLossesBackend.cpython-36m-x86_64-linux-gnu.so

Any suggestions how to make it work? Thanks!

stevenygd commented 4 years ago

Maybe this helps: https://github.com/pytorch/pytorch/issues/6529

AmingWu commented 4 years ago

@cookiew， Could you tell me how much GPU memory the model needs? Thank you

illaMcbender commented 4 years ago

@cookiew Hi!Have you solved the problem? I had exactly the same problem. I'm also using Tesla K80 GPU and installed the dependencies as instructed and got the same error. I checked the suggestions of @stevenygd , but it seems not the problem mentioned in that link. I'm wondering how to solve it. Thank you!

cookiew commented 4 years ago

@illaMcbender k80 has compute compatibility 3.7 (according to this page https://developer.nvidia.com/cuda-gpus#compute). So you need to set the correct CUDA_ARCH parameter in the make file. I.e. add the following line -gencode arch=compute_37,code=sm_37.

This solved the problem for me.

illaMcbender commented 4 years ago

@cookiew It works! Thank you very much!