bigsheep2012
commented
5 years ago Update: Problem solved. It is caused by mismatch between cuda version compiling pytorch and the runtime cuda version. That is, make sure: torch.version.cuda nvcc --version shows the same cuda version.
Hello @sshaoshuai , Thanks for sharing this work. I am getting an error called "undefined symbol: __cudaPopCallConfiguration"
My environment: Pytorch 1.0 Cuda 10.0 GPU driver: 410.78 Python: 3.6 GCC: 5.5
I am guessing the problem might be caused by cuda version mismatch. The full trace log is as followings: python train_and_eval.py --batch_size 8 --epochs 100 --ckpt_save_interval 2 Traceback (most recent call last): File "train_and_eval.py", line 186, in
MODEL = importlib.import_module(args.net) # import network module
File "/home/lin/anaconda3/envs/py36/lib/python3.6/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 994, in _gcd_import
File "", line 971, in _find_and_load
File "", line 955, in _find_and_load_unlocked
File "", line 665, in _load_unlocked
File "", line 678, in exec_module
File "", line 219, in _call_with_frames_removed
File "/media/Lindisk/git_projects/Pointnet2.PyTorch/tools/pointnet2_msg.py", line 3, in
from pointnet2.pointnet2_modules import PointnetFPModule, PointnetSAModuleMSG
File "/media/Lindisk/git_projects/Pointnet2.PyTorch/tools/../pointnet2/pointnet2_modules.py", line 5, in
from . import pointnet2_utils
File "/media/Lindisk/git_projects/Pointnet2.PyTorch/tools/../pointnet2/pointnet2_utils.py", line 7, in
import pointnet2_cuda as pointnet2
ImportError: /home/lin/anaconda3/envs/py36/lib/python3.6/site-packages/pointnet2-0.0.0-py3.6-linux-x86_64.egg/pointnet2_cuda.cpython-36m-x86_64-linux-gnu.so: undefined symbol: __cudaPopCallConfiguration
May I know your environment configurations? or any other idea about why this is happening? Thanks in advance.