search

start-jsk / jsk_apc

Other
36 stars 35 forks source link

Cupy cannot use cuDNN on baxter-c1 #2696

Closed pazeshun closed 5 years ago

pazeshun commented 5 years ago

When I ran my recognition launch, I got the following error:

[ERROR] [1563182895.575925] [/left_hand_camera/fcn_object_segmentation:rosout]: bad callback: <bound method FCNObjectSegmentation._input_cb of <__main__.FCNObjectSegmentation object at 0x7fb87ebc01d0>>
Traceback (most recent call last):
  File "/opt/ros/kinetic/lib/python2.7/dist-packages/rospy/topics.py", line 750, in _invoke_callback
    cb(msg)
  File "/home/pazeshun/ros_mvtk/src/ros_mvtk/grasp_prediction_arc2017/nodes/fcn_object_segmentation.py", line 76, in _input_cb
    self.model(x)
  File "/home/pazeshun/.local/lib/python2.7/site-packages/mvtk/contrib/grasp_prediction_arc2017/models/fcn8s.py", line 76, in __call__
    h = F.relu(self.conv1_1(x))
  File "/usr/local/lib/python2.7/dist-packages/chainer/link.py", line 242, in __call__
    out = forward(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/chainer/links/connection/convolution_2d.py", line 175, in forward
    groups=self.groups)
  File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_2d.py", line 571, in convolution_2d
    y, = fnode.apply(args)
  File "/usr/local/lib/python2.7/dist-packages/chainer/function_node.py", line 263, in apply
    outputs = self.forward(in_data)
  File "/usr/local/lib/python2.7/dist-packages/chainer/function_node.py", line 369, in forward
    return self.forward_gpu(inputs)
  File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_2d.py", line 167, in forward_gpu
    return self._forward_cudnn(x, W, b, y)
  File "/usr/local/lib/python2.7/dist-packages/chainer/functions/connection/convolution_2d.py", line 228, in _forward_cudnn
    auto_tune=auto_tune, tensor_core=tensor_core)
  File "cupy/cudnn.pyx", line 1264, in cupy.cudnn.convolution_forward
  File "cupy/cudnn.pyx", line 32, in cupy.cudnn.get_handle
  File "cupy/cuda/cudnn.pyx", line 752, in cupy.cuda.cudnn.create
  File "cupy/cuda/cudnn.pyx", line 712, in cupy.cuda.cudnn.check_status
CuDNNError: CUDNN_STATUS_NOT_INITIALIZED

When I installed cupy to the user install directory, this error disappeared.

$ dpkg -l cuda
要望=(U)不明/(I)インストール/(R)削除/(P)完全削除/(H)保持
| 状態=(N)無/(I)インストール済/(C)設定/(U)展開/(F)設定失敗/(H)半インストール/(W)トリガ待ち/(T)トリガ保留
|/ エラー?=(空欄)無/(R)要再インストール (状態,エラーの大文字=異常)
||/ 名前                         バージョン          アーキテクチャ      説明
+++-============================-===================-===================-=============================================================
ii  cuda                         9.2.148-1           amd64               CUDA meta-package
$ pip show cupy-cuda92
Name: cupy-cuda92
Version: 5.4.0
Summary: CuPy: NumPy-like API accelerated with CUDA
Home-page: https://docs-cupy.chainer.org/
Author: Seiya Tokui
Author-email: tokui@preferred.jp
License: MIT License
Location: /usr/local/lib/python2.7/dist-packages
Requires: numpy, six, fastrlock
$ pip install cupy-cuda92==5.4.0 --user -I --no-cache-dir
Collecting cupy-cuda92==5.4.0
  Downloading https://files.pythonhosted.org/packages/ac/ae/31ec53a2d7e7c5ea41b58a7b268bbde01add564a8f9fe28fac74ceb07759/cupy_cuda92-5.4.0-cp27-cp27mu-manylinux1_x86_64.whl (317.9MB)
    100% |████████████████████████████████| 317.9MB 13.5MB/s 
Collecting numpy>=1.9.0 (from cupy-cuda92==5.4.0)
  Downloading https://files.pythonhosted.org/packages/1f/c7/198496417c9c2f6226616cff7dedf2115a4f4d0276613bab842ec8ac1e23/numpy-1.16.4-cp27-cp27mu-manylinux1_x86_64.whl (17.0MB)
    100% |████████████████████████████████| 17.0MB 13.4MB/s 
Collecting six>=1.9.0 (from cupy-cuda92==5.4.0)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting fastrlock>=0.3 (from cupy-cuda92==5.4.0)
  Downloading https://files.pythonhosted.org/packages/d4/e6/6d198d91ae20353140563ba32eac2efba236446aa6cf73b2d652d9d9d038/fastrlock-0.4-cp27-cp27mu-manylinux1_x86_64.whl
Installing collected packages: numpy, six, fastrlock, cupy-cuda92
Successfully installed cupy-cuda92-5.4.0 fastrlock-0.4 numpy-1.16.4 six-1.12.0
$ pip show cupy-cuda92
Name: cupy-cuda92
Version: 5.4.0
Summary: CuPy: NumPy-like API accelerated with CUDA
Home-page: https://docs-cupy.chainer.org/
Author: Seiya Tokui
Author-email: tokui@preferred.jp
License: MIT License
Location: /home/pazeshun/.local/lib/python2.7/site-packages
Requires: numpy, six, fastrlock
$ pip uninstall fastrlock
Uninstalling fastrlock-0.4:
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock-0.4.dist-info/INSTALLER
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock-0.4.dist-info/METADATA
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock-0.4.dist-info/RECORD
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock-0.4.dist-info/WHEEL
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock-0.4.dist-info/top_level.txt
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/__init__.pxd
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/__init__.py
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/__init__.pyc
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/_lock.pxi
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/rlock.pxd
  /home/pazeshun/.local/lib/python2.7/site-packages/fastrlock/rlock.so
Proceed (y/n)? y
  Successfully uninstalled fastrlock-0.4
$ pip show fastrlock
Name: fastrlock
Version: 0.4
Summary: Fast, re-entrant optimistic lock implemented in Cython
Home-page: https://github.com/scoder/fastrlock
Author: Stefan Behnel
Author-email: stefan_ml@behnel.de
License: MIT style
Location: /usr/local/lib/python2.7/dist-packages
Requires: 
$ pip uninstall six
Uninstalling six-1.12.0:
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/INSTALLER
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/LICENSE
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/METADATA
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/RECORD
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/WHEEL
  /home/pazeshun/.local/lib/python2.7/site-packages/six-1.12.0.dist-info/top_level.txt
  /home/pazeshun/.local/lib/python2.7/site-packages/six.py
  /home/pazeshun/.local/lib/python2.7/site-packages/six.pyc
Proceed (y/n)? y
  Successfully uninstalled six-1.12.0
$ pip show six
Name: six
Version: 1.12.0
Summary: Python 2 and 3 compatibility utilities
Home-page: https://github.com/benjaminp/six
Author: Benjamin Peterson
Author-email: benjamin@python.org
License: MIT
Location: /usr/local/lib/python2.7/dist-packages
Requires:
pazeshun commented 5 years ago 
sudo pip uninstall cupy
sudo pip uninstall cupy-cuda92
sudo pip install chainer cupy-cuda92 --no-cache-dir -vvv -U

solved problem.