ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 中止 (コアダンプ)

oscarriddle commented 8 years ago

Hi Andersbll,

GTX1080 with CUDA8.0.27 and cudnn 5.1 here. Installed cudarray with cuda back-end. Then try running this project but failed because illegal memory access. LOG:

Traceback (most recent call last): File "neural_artistic_style.py", line 138, in run() File "neural_artistic_style.py", line 130, in run cost = np.mean(net.update()) File "/home/caoxiaolun/Workspace/CNN/neural_artistic_style/neural_artistic_style-master/style_network.py", line 150, in update weight = float(self.style_weights[l]) / norm File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 124, in rdiv return elementwise.divide(other, self) File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 167, in divide return binary(elementwise.div_op, x1, x2, out) File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 87, in binary out = cudarray.empty(array.shape, dtype=out_dtype) File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 246, in empty return ndarray(shape, dtype=dtype) File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 36, in init self._data = ArrayData(self.size, dtype, np_data) File "cudarray/wrap/array_data.pyx", line 16, in cudarray.wrap.array_data.ArrayData.init (./cudarray/wrap/array_data.cpp:1473) File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:824) ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 中止 (コアダンプ)

Besides, if I switch user to root and run, CUDA back-end will be unavailable: "CUDArray: CUDA back-end not available, using NumPy." But normal user will not show up this.

Thanks for your help. oscarriddle

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 935 G /usr/lib/xorg/Xorg 231MiB | | 0 1558 G compiz 109MiB | | 0 1894 G /usr/lib/firefox/firefox 2MiB | +-----------------------------------------------------------------------------+

oscarriddle commented 8 years ago

Hi,

In previous issue: here It seems that this issue is somehow because of out of memory. I checked nvidia-smi and find at peak, python will consume more than 2GB memory.

I just tried to reproduce the image by command: python neural_artistic_style.py --subject images/tuebingen.jpg --style images/starry_night.jpg Not sure why there will be an out of memory issue.

+-----------------------------------------------------------------------------+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=============================================================================| | 0 935 G /usr/lib/xorg/Xorg 279MiB | | 0 1558 G compiz 119MiB | | 0 1894 G /usr/lib/firefox/firefox 2MiB | | 0 15425 C python 2395MiB | +-----------------------------------------------------------------------------+

oscarriddle commented 8 years ago

Continue. I tried to step by step check whether I lost something so now cudarray cannot work. 1) By running CUDA 8.0RC samples, the CUDA 8.0RC is working. check 2) cuDNN is installed correctly. check 3) After including export path and env variable CUDNN_ENABLED=1 into /etc/profile. Now cudarray is installed and can be imported. check 4) I tried cudarray examples/test.py then got below log:

$python test.py True . . . True True True False True True True False (3.0232945639326929, 3.0016344) (10.010378811733982, 9.9944296) (0.49997663711219476, 0.50012529) (0.28865082930213165, 0.28864652) (10.019171134185873, 10.004223) (11.540598502938147, 11.549608) Traceback (most recent call last): File "test.py", line 494, in run() File "test.py", line 488, in run test_reduce() File "test.py", line 347, in test_reduce print(np.allclose(c_np, np.array(c_ca))) File "/usr/local/lib/python2.7/dist-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 42, in array self._data.to_numpy(np_array) File "cudarray/wrap/array_data.pyx", line 24, in cudarray.wrap.array_data.ArrayData.to_numpy (./cudarray/wrap/array_data.cpp:1593) File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:824) ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 中止 (コアダンプ)

This also showd that the cudarray is not working. Not sure whether because it cannot utilize the cuDNN and CUDA, so any memory allocation(even very tiny) will exceed the limitation. Still not sure how to solve it.

atonn commented 8 years ago

just wanted to note that I'm having the same issue on a GTX1080, which is funny because it worked a month or so ago with a different GTX1080. Other deep learning frameworks are working fine right now, will post again once I find out more, probably will go with a system reinstall first

nudtfuruigang commented 7 years ago

have the same issue on a GTX850M, which makes me disappointed

wlxyhy commented 7 years ago

the same problem, is there anyone know how to solve it?

albertyou2 commented 7 years ago

@andersbll @oscarriddle I also met this problem.here is the log:

Traceback (most recent call last): File "neural_artistic_style.py", line 138, in run() File "neural_artistic_style.py", line 130, in run cost = np.mean(net.update()) File "/home/sojoyoo/neutral_style_adv_2/style_network.py", line 150, in update weight = float(self.style_weights[l]) / norm File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 124, in rdiv return elementwise.divide(other, self) File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 167, in divide return binary(elementwise.div_op, x1, x2, out) File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/elementwise.py", line 87, in binary out = cudarray.empty(array.shape, dtype=out_dtype) File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 246, in empty return ndarray(shape, dtype=dtype) File "/home/sojoyoo/anaconda2/lib/python2.7/site-packages/cudarray-0.1.dev0-py2.7-linux-x86_64.egg/cudarray/cudarray.py", line 36, in init self._data = ArrayData(self.size, dtype, np_data) File "cudarray/wrap/array_data.pyx", line 16, in cudarray.wrap.array_data.ArrayData.init (./cudarray/wrap/array_data.cpp:1465) File "cudarray/wrap/cudart.pyx", line 12, in cudarray.wrap.cudart.cudaCheck (./cudarray/wrap/cudart.cpp:816) ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 已放弃 (核心已转储)

env is ： Ubuntu 16.04 cuda 8.0,cudnn 5.0, gforce gtx 1070(8G memory)

andersbll / neural_artistic_style

ValueError: an illegal memory access was encountered terminate called after throwing an instance of 'std::runtime_error' what(): ./include/cudarray/common.hpp:95: an illegal memory access was encountered 中止 (コアダンプ) #48