search

akanazawa / cmr

Project repo for Learning Category-Specific Mesh Reconstruction from Image Collections
MIT License
474 stars 84 forks source link

cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins #4

Closed akjagadish closed 5 years ago

akjagadish commented 5 years ago

When you run the demo: python -m cmr.demo --name bird_net --num_train_epoch 500 --img_path cmr/demo_data/img1.jpg

I am getting this error:

/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/h5py/init.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type. from ._conv import register_converters as _register_converters Setting up model.. loading /home/ubuntu/akshay/cmr/nnutils/../cachedir/snapshots/bird_net/pred_net_500.pth.. Traceback (most recent call last): File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main "main", fname, loader, pkg_name) File "/usr/lib/python2.7/runpy.py", line 72, in _run_code exec code in run_globals File "/home/ubuntu/akshay/cmr/demo.py", line 119, in app.run(main) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/absl/app.py", line 274, in run _run_main(main, argv) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/absl/app.py", line 238, in _run_main sys.exit(main(argv)) File "/home/ubuntu/akshay/cmr/demo.py", line 108, in main outputs = predictor.predict(batch) File "cmr/nnutils/predictor.py", line 110, in predict self.forward() File "cmr/nnutils/predictor.py", line 149, in forward self.cam_pred) File "cmr/nnutils/nmr.py", line 183, in forward return Render(self.renderer)(verts, faces) File "cmr/nnutils/nmr.py", line 114, in forward masks = self.renderer.forward_mask(vs, fs) File "cmr/nnutils/nmr.py", line 50, in forward_mask self.masks = self.renderer.render_silhouettes(self.vertices, self.faces) File "build/bdist.linux-x86_64/egg/neural_renderer/renderer.py", line 38, in render_silhouettes File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/functions/array/concat.py", line 90, in concat y, = Concat(axis).apply(xs) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/function_node.py", line 245, in apply outputs = self.forward(in_data) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/chainer/functions/array/concat.py", line 44, in forward return xp.concatenate(xs, self.axis), File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/manipulation/join.py", line 49, in concatenate return core.concatenate_method(tup, axis) File "cupy/core/core.pyx", line 2439, in cupy.core.core.concatenate_method File "cupy/core/core.pyx", line 2482, in cupy.core.core.concatenate_method File "cupy/core/core.pyx", line 2533, in cupy.core.core.concatenate File "cupy/core/core.pyx", line 1630, in cupy.core.core.ndarray.setitem File "cupy/core/core.pyx", line 3101, in cupy.core.core._scatter_op File "cupy/core/elementwise.pxi", line 823, in cupy.core.core.ufunc.call File "cupy/util.pyx", line 39, in cupy.util.memoize.decorator.ret File "cupy/core/elementwise.pxi", line 622, in cupy.core.core._get_ufunc_kernel File "cupy/core/elementwise.pxi", line 33, in cupy.core.core._get_simple_elementwise_kernel File "cupy/core/carray.pxi", line 170, in cupy.core.core.compile_with_cache File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 123, in compile_with_cache base = _preprocess('', options, arch) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 86, in _preprocess result = prog.compile(options) File "/home/ubuntu/akshay/cmr/venv_cmr/local/lib/python2.7/site-packages/cupy/cuda/compiler.py", line 233, in compile raise CompileException(log, self.src, self.name, options) cupy.cuda.compiler.CompileException: nvrtc: error: failed to load builtins

I tried the solution mentioned in this: https://groups.google.com/forum/#!topic/chainer-jp/GKNe5KY_fm0 yet it is not working

akjagadish commented 5 years ago

The method suggested in https://groups.google.com/forum/#!topic/chainer-jp/GKNe5KY_fm0 worked. Had multiple versions of CUDA installed, need to figure out the right version.

viralbthakar commented 5 years ago

Hi @akshaykjagadish I am facing the same problem. Can you suggest the exact steps you followed to solve the problem? Thank you

akjagadish commented 5 years ago

Hi, it has been some time since I solved this issue but I figure you just need to find out the right CUDA version (I had multiple versions on my AWS instance) and set path accordingly.

PenroseWang commented 5 years ago

Hi @akshaykjagadish , I am also facing the same problem. Do you still remember which CUDA version are you using? I currently only have CUDA 9.0 installed on my Linux, but still have the CompileException.

And hi @viralbthakar , did you solve the problem later?

Thank you all.

charliememory commented 5 years ago

I use the new version of neural_render git clone git@github.com:charliememory/neural_renderer.git and install the pre-built cupy with corresponding cuda version number, e.g. pip install cupy-cuda92. Then, I can avoid this issue.

nikhilaravi commented 5 years ago

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error: RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

BornInWater commented 5 years ago

@charliememory I used neural renderer from the repo that you mentioned and have cuda 9 and hence used the same command but while installing neural renderer I get the below error:

Traceback (most recent call last): File "setup.py", line 3, in import neural_renderer File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/init.py", line 1, in from cross import cross File "/home/nsajjan/WORK/Capstone/cmr/external/neural_renderer/neural_renderer/cross.py", line 2, in import cupy as cp File "/home/nsajjan/WORK/Capstone/cmr/venv_cmr/lib/python2.7/site-packages/cupy/init.py", line 7, in from cupy import _version ImportError: cannot import name _version

Any idea why I am getting this?

charliememory commented 5 years ago

@nikhilaravi I did not have such issue. Maybe you can check whether the CUDA is accessible for other packages first to make sure the problem comes from chainer.

charliememory commented 5 years ago

@BornInWater In my case, I modified the setup.py file. I used cupy-cuda92, maybe you can try cupy-cuda9 instead.

BornInWater commented 5 years ago

@charliememory Tried that. Not working. Same error.

ZilinZhou1995 commented 5 years ago

@charliememory did you use chainer 3.3.0 or the latest version? With chainer==3.3.0 I keep getting an error: RuntimeError: CUDA environment is not correctly set up (see https://github.com/chainer/chainer#installation).cannot import name sqrt_fixed I checked chainer.backends.cuda.available and it returned False. With the new version of chainer, I get the nvrtc error above even though I have set the CUDA_PATH and LD_LIBRARY_PATH as instructed in the cupy documentation.

Have you slove this problem？

BornInWater commented 5 years ago

@ZilinZhou1995 , No, I could not solve it.

BornInWater commented 5 years ago

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works. After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest. Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0. Hope this helps!

BornInWater commented 5 years ago

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

QHan1st commented 4 years ago

@charliememory You are installing the newest version of NMR right? Were you able to train the model properly with the newest version of NMR? For me one of the losses was much higher than the cmr with the prescribed nmr. You can see the conversation here

Have you run the project successfully?

EAST-J commented 4 years ago

@nikhilaravi , @ZilinZhou1995 Its working for me now.

I have CUDA 9. I first installed chainer 5.2.0 version and cupy 5.2.0[pip install cupy-cuda90] version separately. Then installed neural mesh renderer and Perceptual similarity in the external directory. But in the original script, branch v1.1.0 of neural mesh renderer gets installed. This was not working for me, so I installed the most recent one. That works. After that, in the requirements.txt file, I removed chainer and cupy and just did pip install -r requirements.txt for the rest. Also, I had to switch to Torch 1.0 version as I think torch 0.3 does not support CUDA 9.0. Hope this helps!

Have you run the project successfully? I have CUDA9.2 and use Torch1.0 but it has errors that says a value of type "const float " cannot be used to initialize an entity of type "float "

ShougangShen commented 4 years ago

python2.7, CUDA8.0, chainer 3.3.0, cupy 2.3.0, pytorch 0.3.1 works for me. Before running the demo command, I run: export PATH="$PATH:/usr/local/cuda-8.0/bin" ， export LD_LIBRARY_PATH="/usr/local/cuda-8.0/lib64" and reinstall the chainer and cupy.