Closed dmitrysharkov closed 7 years ago
UPDATE: i have the same unsupported operation issue with opencl0:0:
THEANO_FLAGS=mode=FAST_RUN,device=opencl0:0,floatX=float32 python -c "import theano"
THEANO_FLAGS=mode=FAST_RUN,device=opencl0:0,floatX=float32 python -c "import theano"
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "/Users/dimson/anaconda/envs/tt/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 164, in <module>
use(config.device)
File "/Users/dimson/anaconda/envs/tt/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 151, in use
init_dev(device)
File "/Users/dimson/anaconda/envs/tt/lib/python2.7/site-packages/theano/gpuarray/__init__.py", line 100, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpu/blas.pyx", line 129, in pygpu.blas.gemm (pygpu/blas.c:3354)
File "pygpu/blas.pyx", line 44, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2011)
GpuArrayException: ('Unsupported operation', 5)
so looks like it's not a GPU problem...
UPDATE 2: When I run tests it throws this 'Unsupported operation' on the following lines:
.... File "pygpu/blas.pyx", line 63, in pygpu.blas.dot File "pygpu/blas.pyx", line 26, in pygpu.blas.pygpu_blas_rdot ... File "pygpu/blas.pyx", line 90, in pygpu.blas.gemv File "pygpu/blas.pyx", line 35, in pygpu.blas.pygpu_blas_rgemv ... File "pygpu/blas.pyx", line 129, in pygpu.blas.gemm File "pygpu/blas.pyx", line 44, in pygpu.blas.pygpu_blas_rgemm
File "pygpu/blas.pyx", line 145, in pygpu.blas.ger File "pygpu/blas.pyx", line 52, in pygpu.blas.pygpu_blas_rger
it happens for all tests of these functions.
I'm nube in Python but it seems the the following functions generate it:
GpuArray_rdot, GpuArray_rgemv, GpuArrayException, GpuArray_rger
UPDATE 3: it seems like GpuArray_ISONESEGMENT always return false
or gpublas_setup alwasy return GA_UNSUPPORTED_ERROR because blas_ops beacuse ctx->blas_ops is NULL
OpenCL support in libgpuarray work, but we don't have good support for it in Theano. You will miss operation in Theano in OpenCL. If you aren't ready to help us make it work, I would recommand that you don't use it. It isn't ready.
In particular, for your error, the problem is that libgpuarray don't find a blas for OpenCL. You should install CLBlas of CLBlast.
On Tue, Apr 11, 2017 at 3:34 PM Dmitry Sharkov notifications@github.com wrote:
UPDATE 3: it seems like GpuArray_ISONESEGMENT always return false
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/402#issuecomment-293376318, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-wc1WxMX2Gi41BMCpMY6TfKHKFc7ks5ru9W4gaJpZM4M6ela .
wow! thanks for so fast reply :) i'm not sure if i'm able to help, because i've seen this code first time in my life tonight :D i study deep learning now and wanted to make it work faster on my comp. I can do something only if you give some very specific task ;)
We aren't able to do so now.
On Tue, Apr 11, 2017 at 4:31 PM Dmitry Sharkov notifications@github.com wrote:
wow! thanks for so fast reply :) i'm not sure if i'm able to help, because i've seen this code first time in my life tonight :D i study deep learning now and wanted to make it work faster on my comp. I can do something only if you give some very specific task ;)
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/402#issuecomment-293391068, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-_ll0BCOX-tO0iu-atM2dDivcahBks5ru-MUgaJpZM4M6ela .
I'm the exact same issue, all of a sudden, on a machine that previously worked without any problem. It's a windows machine, with packages installed in a venv with anaconda.
I face this issue with versions 0.6.2 and 0.6.4 for libgpuarray and pygpu (which is the version required by Theano).
Error is very easy to reproduce, just by running python -c "import theano"
$ (theano_py35) python -c "import theano"
Can not use cuDNN on context None: Disabled by dnn.enabled flag
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module>
use(config.device)
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use
init_dev(device)
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 100, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpu\blas.pyx", line 129, in pygpu.blas.gemm (pygpu/blas.c:3350)
File "pygpu\blas.pyx", line 44, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2007)
pygpu.gpuarray.GpuArrayException: (b'Unsupported operation', 5)
(theano_py35) $ conda list
cycler 0.10.0 py35_0
decorator 4.1.2 <pip>
ExifRead 2.1.2 <pip>
h5py 2.7.0 <pip>
icu 57.1 vc14_0 [vc14]
jpeg 9b vc14_0 [vc14]
Keras 1.2.2 <pip>
libgpuarray 0.6.2 vc14_0 [vc14]
libpng 1.6.27 vc14_0 [vc14]
libpython 2.0 py35_0
m2w64-binutils 2.25.1 5
m2w64-bzip2 1.0.6 6
m2w64-crt-git 5.0.0.4636.2595836 2
m2w64-gcc 5.3.0 6
m2w64-gcc-ada 5.3.0 6
m2w64-gcc-fortran 5.3.0 6
m2w64-gcc-libgfortran 5.3.0 6
m2w64-gcc-libs 5.3.0 7
m2w64-gcc-libs-core 5.3.0 7
m2w64-gcc-objc 5.3.0 6
m2w64-gmp 6.1.0 2
m2w64-headers-git 5.0.0.4636.c0ad18a 2
m2w64-isl 0.16.1 2
m2w64-libiconv 1.14 6
m2w64-libmangle-git 5.0.0.4509.2e5a9a2 2
m2w64-libwinpthread-git 5.0.0.4634.697f757 2
m2w64-make 4.1.2351.a80a8b8 2
m2w64-mpc 1.0.3 3
m2w64-mpfr 3.1.4 4
m2w64-pkg-config 0.29.1 2
m2w64-toolchain 5.3.0 7
m2w64-tools-git 5.0.0.4592.90b8472 2
m2w64-windows-default-manifest 6.4 3
m2w64-winpthreads-git 5.0.0.4634.697f757 2
m2w64-zlib 1.2.8 10
mako 1.0.6 py35_0
markupsafe 0.23 py35_2
matplotlib 2.0.2 np113py35_0
mkl 2017.0.3 0
mkl-service 1.1.2 py35_3
msys2-conda-epoch 20160418 1
munch 2.2.0 <pip>
networkx 1.11 <pip>
neural-image-analogies 0.2.4 <pip>
nose 1.3.7 py35_1
numpy 1.13.1 py35_0
olefile 0.44 <pip>
openssl 1.0.2l vc14_0 [vc14]
Pillow 4.2.1 <pip>
pip 9.0.1 py35_1
pygpu 0.6.2 py35_0
pyparsing 2.2.0 py35_0
pyqt 5.6.0 py35_2
python 3.5.3 3
python-dateutil 2.6.1 py35_0
pytz 2017.2 py35_0
PyWavelets 0.5.2 <pip>
PyYAML 3.12 <pip>
qt 5.6.2 vc14_6 [vc14]
scikit-image 0.13.0 <pip>
scikit-learn 0.18.2 <pip>
scipy 0.19.1 np113py35_0
setuptools 27.2.0 py35_1
sip 4.18 py35_0
six 1.10.0 py35_1
theano 0.9.0 py35_0
tk 8.5.18 vc14_0 [vc14]
vs2015_runtime 14.0.25420 0
wheel 0.29.0 py35_0
zlib 1.2.8 vc14_3 [vc14]
With version 0.6.8 the error becomes different (but I guess it is due to incompatibility with Theano more than anything else)
(theano_py35) $ python -c "import theano"
Can not use cuDNN on context None: Disabled by dnn.enabled flag
ERROR (theano.gpuarray): Could not initialize pygpu, support disabled
Traceback (most recent call last):
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 164, in <module>
use(config.device)
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 151, in use
init_dev(device)
File "C:\Users\***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\__init__.py", line 100, in init_dev
pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True)
File "pygpu\blas.pyx", line 149, in pygpu.blas.gemm (pygpu/blas.c:3535)
File "pygpu\blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2025)
pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)
Any idea on how to fix that ? I insist on the fact that the machine previously worked, and no updates were made, neither to CUDA nor to the virtual env packages. So I'm guessing all it would take is a bit of cleanup somewhere.
Thanks for you help
Can you try with libgpuarray 0.6.9?
Windows have the bad habit the slow GPU kernel can "timeout". This can happen randomly due to other stuff running at the same time on the GPU can slow down the computation enough to make it pass the threashold. So even with an update, if this is your problem, the only work around is to mess with windows registry to tell it to raise the timeout threashold.
On Tue, Aug 22, 2017 at 4:46 AM Rémi Bèges notifications@github.com wrote:
I'm the exact same issue, all of a sudden, on a machine that previously worked without any problem. It's a windows machine, with packages installed in a venv with anaconda.
I face this issue with versions 0.6.2 and 0.6.4 for libgpuarray and pygpu (which is the version required by Theano).
Error is very easy to reproduce, just by running python -c "import theano"
$ (theano_py35) python -c "import theano" Can not use cuDNN on context None: Disabled by dnn.enabled flag ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users*\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init__.py", line 164, in
use(config.device) File "C:\Users* \Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init.py", line 151, in use init_dev(device) File "C:\Users***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\init__.py", line 100, in init_dev pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) File "pygpu\blas.pyx", line 129, in pygpu.blas.gemm (pygpu/blas.c:3350) File "pygpu\blas.pyx", line 44, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2007) pygpu.gpuarray.GpuArrayException: (b'Unsupported operation', 5)(theano_py35) $ conda list cycler 0.10.0 py35_0 decorator 4.1.2
ExifRead 2.1.2 h5py 2.7.0 icu 57.1 vc14_0 [vc14] jpeg 9b vc14_0 [vc14] Keras 1.2.2 libgpuarray 0.6.2 vc14_0 [vc14] libpng 1.6.27 vc14_0 [vc14] libpython 2.0 py35_0 m2w64-binutils 2.25.1 5 m2w64-bzip2 1.0.6 6 m2w64-crt-git 5.0.0.4636.2595836 2 m2w64-gcc 5.3.0 6 m2w64-gcc-ada 5.3.0 6 m2w64-gcc-fortran 5.3.0 6 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gcc-objc 5.3.0 6 m2w64-gmp 6.1.0 2 m2w64-headers-git 5.0.0.4636.c0ad18a 2 m2w64-isl 0.16.1 2 m2w64-libiconv 1.14 6 m2w64-libmangle-git 5.0.0.4509.2e5a9a2 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-make 4.1.2351.a80a8b8 2 m2w64-mpc 1.0.3 3 m2w64-mpfr 3.1.4 4 m2w64-pkg-config 0.29.1 2 m2w64-toolchain 5.3.0 7 m2w64-tools-git 5.0.0.4592.90b8472 2 m2w64-windows-default-manifest 6.4 3 m2w64-winpthreads-git 5.0.0.4634.697f757 2 m2w64-zlib 1.2.8 10 mako 1.0.6 py35_0 markupsafe 0.23 py35_2 matplotlib 2.0.2 np113py35_0 mkl 2017.0.3 0 mkl-service 1.1.2 py35_3 msys2-conda-epoch 20160418 1 munch 2.2.0 networkx 1.11 neural-image-analogies 0.2.4 nose 1.3.7 py35_1 numpy 1.13.1 py35_0 olefile 0.44 openssl 1.0.2l vc14_0 [vc14] Pillow 4.2.1 pip 9.0.1 py35_1 pygpu 0.6.2 py35_0 pyparsing 2.2.0 py35_0 pyqt 5.6.0 py35_2 python 3.5.3 3 python-dateutil 2.6.1 py35_0 pytz 2017.2 py35_0 PyWavelets 0.5.2 PyYAML 3.12 qt 5.6.2 vc14_6 [vc14] scikit-image 0.13.0 scikit-learn 0.18.2 scipy 0.19.1 np113py35_0 setuptools 27.2.0 py35_1 sip 4.18 py35_0 six 1.10.0 py35_1 theano 0.9.0 py35_0 tk 8.5.18 vc14_0 [vc14] vs2015_runtime 14.0.25420 0 wheel 0.29.0 py35_0 zlib 1.2.8 vc14_3 [vc14] With version 0.6.8 the error becomes different (but I guess it is due to incompatibility with Theano more than anything else)
(theano_py35) $ python -c "import theano" Can not use cuDNN on context None: Disabled by dnn.enabled flag ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users*\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init__.py", line 164, in
use(config.device) File "C:\Users* \Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init.py", line 151, in use init_dev(device) File "C:\Users***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\init__.py", line 100, in init_dev pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) File "pygpu\blas.pyx", line 149, in pygpu.blas.gemm (pygpu/blas.c:3535) File "pygpu\blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2025) pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)Any idea on how to fix that ? I insist on the fact that the machine previously worked, and no updates were made, neither to CUDA nor to the virtual env packages. So I'm guessing all it would take is a bit of cleanup somewhere.
Thanks for you help
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/402#issuecomment-323960772, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-2aLSgVkrzf3SexCygS9sMdrakvnks5sapVLgaJpZM4M6ela .
Note, the your macbook, you probably don't have an NVIDIA GPU. The openCL back-end isn't in a usable state. So just use the Theano "cpu" back-end.
On Tue, Aug 22, 2017 at 11:31 AM Frédéric Bastien < frederic.bastien@gmail.com> wrote:
Can you try with libgpuarray 0.6.9?
Windows have the bad habit the slow GPU kernel can "timeout". This can happen randomly due to other stuff running at the same time on the GPU can slow down the computation enough to make it pass the threashold. So even with an update, if this is your problem, the only work around is to mess with windows registry to tell it to raise the timeout threashold.
On Tue, Aug 22, 2017 at 4:46 AM Rémi Bèges notifications@github.com wrote:
I'm the exact same issue, all of a sudden, on a machine that previously worked without any problem. It's a windows machine, with packages installed in a venv with anaconda.
I face this issue with versions 0.6.2 and 0.6.4 for libgpuarray and pygpu (which is the version required by Theano).
Error is very easy to reproduce, just by running python -c "import theano"
$ (theano_py35) python -c "import theano" Can not use cuDNN on context None: Disabled by dnn.enabled flag ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users*\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init__.py", line 164, in
use(config.device) File "C:\Users* \Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init.py", line 151, in use init_dev(device) File "C:\Users***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\init__.py", line 100, in init_dev pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) File "pygpu\blas.pyx", line 129, in pygpu.blas.gemm (pygpu/blas.c:3350) File "pygpu\blas.pyx", line 44, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2007) pygpu.gpuarray.GpuArrayException: (b'Unsupported operation', 5)(theano_py35) $ conda list cycler 0.10.0 py35_0 decorator 4.1.2
ExifRead 2.1.2 h5py 2.7.0 icu 57.1 vc14_0 [vc14] jpeg 9b vc14_0 [vc14] Keras 1.2.2 libgpuarray 0.6.2 vc14_0 [vc14] libpng 1.6.27 vc14_0 [vc14] libpython 2.0 py35_0 m2w64-binutils 2.25.1 5 m2w64-bzip2 1.0.6 6 m2w64-crt-git 5.0.0.4636.2595836 2 m2w64-gcc 5.3.0 6 m2w64-gcc-ada 5.3.0 6 m2w64-gcc-fortran 5.3.0 6 m2w64-gcc-libgfortran 5.3.0 6 m2w64-gcc-libs 5.3.0 7 m2w64-gcc-libs-core 5.3.0 7 m2w64-gcc-objc 5.3.0 6 m2w64-gmp 6.1.0 2 m2w64-headers-git 5.0.0.4636.c0ad18a 2 m2w64-isl 0.16.1 2 m2w64-libiconv 1.14 6 m2w64-libmangle-git 5.0.0.4509.2e5a9a2 2 m2w64-libwinpthread-git 5.0.0.4634.697f757 2 m2w64-make 4.1.2351.a80a8b8 2 m2w64-mpc 1.0.3 3 m2w64-mpfr 3.1.4 4 m2w64-pkg-config 0.29.1 2 m2w64-toolchain 5.3.0 7 m2w64-tools-git 5.0.0.4592.90b8472 2 m2w64-windows-default-manifest 6.4 3 m2w64-winpthreads-git 5.0.0.4634.697f757 2 m2w64-zlib 1.2.8 10 mako 1.0.6 py35_0 markupsafe 0.23 py35_2 matplotlib 2.0.2 np113py35_0 mkl 2017.0.3 0 mkl-service 1.1.2 py35_3 msys2-conda-epoch 20160418 1 munch 2.2.0 networkx 1.11 neural-image-analogies 0.2.4 nose 1.3.7 py35_1 numpy 1.13.1 py35_0 olefile 0.44 openssl 1.0.2l vc14_0 [vc14] Pillow 4.2.1 pip 9.0.1 py35_1 pygpu 0.6.2 py35_0 pyparsing 2.2.0 py35_0 pyqt 5.6.0 py35_2 python 3.5.3 3 python-dateutil 2.6.1 py35_0 pytz 2017.2 py35_0 PyWavelets 0.5.2 PyYAML 3.12 qt 5.6.2 vc14_6 [vc14] scikit-image 0.13.0 scikit-learn 0.18.2 scipy 0.19.1 np113py35_0 setuptools 27.2.0 py35_1 sip 4.18 py35_0 six 1.10.0 py35_1 theano 0.9.0 py35_0 tk 8.5.18 vc14_0 [vc14] vs2015_runtime 14.0.25420 0 wheel 0.29.0 py35_0 zlib 1.2.8 vc14_3 [vc14] With version 0.6.8 the error becomes different (but I guess it is due to incompatibility with Theano more than anything else)
(theano_py35) $ python -c "import theano" Can not use cuDNN on context None: Disabled by dnn.enabled flag ERROR (theano.gpuarray): Could not initialize pygpu, support disabled Traceback (most recent call last): File "C:\Users*\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init__.py", line 164, in
use(config.device) File "C:\Users* \Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray__init.py", line 151, in use init_dev(device) File "C:\Users***\Miniconda3\envs\theano_py35\lib\site-packages\theano\gpuarray\init__.py", line 100, in init_dev pygpu.blas.gemm(0, tmp, tmp, 0, tmp, overwrite_c=True) File "pygpu\blas.pyx", line 149, in pygpu.blas.gemm (pygpu/blas.c:3535) File "pygpu\blas.pyx", line 47, in pygpu.blas.pygpu_blas_rgemm (pygpu/blas.c:2025) pygpu.gpuarray.GpuArrayException: (b'Missing Blas library', 5)Any idea on how to fix that ? I insist on the fact that the machine previously worked, and no updates were made, neither to CUDA nor to the virtual env packages. So I'm guessing all it would take is a bit of cleanup somewhere.
Thanks for you help
— You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/402#issuecomment-323960772, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-2aLSgVkrzf3SexCygS9sMdrakvnks5sapVLgaJpZM4M6ela .
@nouiz thanks a lot for the feedback
The key TdrDelay
in HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\GraphicsDrivers
(see here ) was set to 10 seconds. I will try increasing it but I'm not sure that's the problem here because the import crashes after 1 or 2 seconds.
With pygpu 0.6.9
, the error is the same as 0.6.8
but different from 0.6.2
/0.6.4
(and the original .
With 0.6.9
/0.6.8
it's complaining about missing Blas library. I though the mkl-service package would solve that. Any idea how to fix that ?
Thanks again
So I reinstalled completely Cuda, and the problem went away. Most likely corrupted files. I happened to kill my python programs with Ctrl
+C
so it might be why files got corrupted. Anyway, I'll stop hijacking this thread now, thanks again for the feedback.
Just for those that want a solution for the initial problem: the OpenCL blas libraries are missing. CLBlast or clBLAS are required to provide this.
Hello, can somebody help me or tell me what i'm doing wrong?
I can't use GPU. for some reason it says 'Unsupported operation' (both python 2.7 and 3.6) I have MacPro. OS Sierra 10.12.4 GPU: Intel Iris Pro Theano 0.9.0 lingpuarray built from git
more details below:
when i deck pyguy init it seems to work (see below)
$> python -c "import pygpu; print pygpu.init('opencl0:1').devname"
Iris Pro