Theano / libgpuarray

Library to manipulate tensors on the GPU.
Other
189 stars 95 forks source link

Tests Fail on AMD OpenCL device #247

Open JonnoFTW opened 8 years ago

JonnoFTW commented 8 years ago

I compiled and tested the latest libgpuarray on Ubuntu 14.04 with the pygpu package. When I run: $ DEVICE="opencl0:0" python -c "import pygpu;pygpu.test()", I get the following errors:

pygpu is installed in /home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu
NumPy version 1.11.1
NumPy relaxed strides checking option: False
NumPy is installed in /home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy
Python version 2.7.12 (default, Jul 19 2016, 15:29:32) [GCC 4.8.4]
nose version 1.3.7
*** Testing for Oland
.......
======================================================================
ERROR: pygpu.tests.test_reduction.test_red_big_array('float32', (2000, 30, 100), [True, False, False])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_reduction.py", line 29, in red_array_sum
    c, g = gen_gpuarray(shape, dtype, ctx=context)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 160, in gen_gpuarray
    assert numpy.allclose(a, numpy.asarray(b)), (a, numpy.asarray(b))
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17346)
  File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6114)
GpuArrayException: Memory object allocation failure

======================================================================
ERROR: pygpu.tests.test_reduction.test_red_big_array('float32', (2000, 30, 100), [True, False, True])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_reduction.py", line 29, in red_array_sum
    c, g = gen_gpuarray(shape, dtype, ctx=context)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 160, in gen_gpuarray
    assert numpy.allclose(a, numpy.asarray(b)), (a, numpy.asarray(b))
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17346)
  File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6114)
GpuArrayException: Memory object allocation failure

======================================================================
ERROR: pygpu.tests.test_reduction.test_red_big_array('float32', (2000, 30, 100), [False, True, True])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_reduction.py", line 29, in red_array_sum
    c, g = gen_gpuarray(shape, dtype, ctx=context)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 160, in gen_gpuarray
    assert numpy.allclose(a, numpy.asarray(b)), (a, numpy.asarray(b))
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17346)
  File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6114)
GpuArrayException: Memory object allocation failure

======================================================================
ERROR: pygpu.tests.test_reduction.test_red_big_array('float32', (2000, 30, 100), [False, True, False])
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_reduction.py", line 29, in red_array_sum
    c, g = gen_gpuarray(shape, dtype, ctx=context)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 160, in gen_gpuarray
    assert numpy.allclose(a, numpy.asarray(b)), (a, numpy.asarray(b))
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy/core/numeric.py", line 482, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1320, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17346)
  File "pygpu/gpuarray.pyx", line 347, in pygpu.gpuarray.array_read (pygpu/gpuarray.c:6114)
GpuArrayException: Memory object allocation failure

======================================================================
FAIL: pygpu.tests.test_elemwise.test_elemwise_f16(<built-in function add>, 'float16', 'float16', (50,))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_elemwise.py", line 72, in elemwise2_ops_array
    assert numpy.allclose(out_c, numpy.asarray(out_g))
AssertionError

======================================================================
FAIL: pygpu.tests.test_elemwise.test_elemwise_f16(<built-in function iadd>, 'float16', 'float16', (50,))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_elemwise.py", line 95, in ielemwise2_ops_array
    assert numpy.allclose(out_c, numpy.asarray(out_g), atol=1e-6)
AssertionError

----------------------------------------------------------------------
Ran 6904 tests in 67.348s
FAILED (errors=4, failures=2)

Are these errors serious? Is it a problem with the code or have I built something incorrectly?

abergeron commented 8 years ago

The memory errors indicate something is wrong at runtime. I don't know what.

The errors in float16, indicate that there probably is a problem with float16 support. It hasn't been tested much with the OpenCL backend so it would not surprise me that there is a problem.

nouiz commented 8 years ago

Just to tell, we don't have time to finish OpenCL support. There is more work needed then this. We need people from outside the current team to come and help us make it to have it working.

Are you interrested?

On Mon, Sep 5, 2016 at 12:18 PM, abergeron notifications@github.com wrote:

The memory errors indicate something is wrong at runtime. I don't know what.

The errors in float16, indicate that there probably is a problem with float16 support. It hasn't been tested much with the OpenCL backend so it would not surprise me that there is a problem.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/247#issuecomment-244780509, or mute the thread https://github.com/notifications/unsubscribe-auth/AALC-7MMNzZxfkznlt_hBtoXEKzpkkdgks5qnEDNgaJpZM4J0rSC .

JonnoFTW commented 8 years ago

As much as I'd like to hope to get into GPGPU programming, I have too much other stuff going in my life at the moment. I can try figuring out what's causing these tests to fail on my machine though.

JonnoFTW commented 8 years ago

I'll also add that more tests fail if you use a numpy compiled Intel MKL. I've pasted the errors below (they're repeated for all the tested types):

======================================================================
ERROR: pygpu.tests.test_tools.test_check_args_collapse_2
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_tools.py", line 67, in test_check_args_collapse_2
    offseted_inner=True)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 160, in gen_gpuarray
    assert numpy.allclose(a, numpy.asarray(b)), (a, numpy.asarray(b))
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/numpy-1.12.0.dev0+cb7f407-py2.7-linux-x86_64.egg/numpy/core/numeric.py", line 531, in asarray
    return array(a, dtype, copy=False, order=order)
  File "pygpu/gpuarray.pyx", line 1572, in pygpu.gpuarray.GpuArray.__array__ (pygpu/gpuarray.c:20224)
  File "pygpu/gpuarray.pyx", line 1314, in pygpu.gpuarray.pygpu_as_ndarray (pygpu/gpuarray.c:17283)
  File "pygpu/gpuarray.pyx", line 652, in pygpu.gpuarray.pygpu_copy (pygpu/gpuarray.c:9262)
  File "pygpu/gpuarray.pyx", line 359, in pygpu.gpuarray.array_copy (pygpu/gpuarray.c:6278)
GpuArrayException: Unaligned array

======================================================================
FAIL: pygpu.tests.test_gpu_ndarray.test_shape(((), (1,)), False, 'c')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_gpu_ndarray.py", line 436, in shape_
    assert ac.strides == ag.strides, (ac.strides, ag.strides)
AssertionError: ((9223372036854775807,), (4,))

======================================================================
FAIL: pygpu.tests.test_gpu_ndarray.test_shape(((4, 3), (-1, 12)), False, 'c', 'f')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_gpu_ndarray.py", line 447, in reshape
    assert outc.strides == outg.strides
AssertionError

======================================================================
FAIL: pygpu.tests.test_gpu_ndarray.test_transpose((1, 2, 3, 4), [1, 2, 3, 0], False, -2, 'c')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_gpu_ndarray.py", line 484, in transpose_perm
    check_all(rg, rc)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 103, in check_all
    check_meta(x, y)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 99, in check_meta
    check_flags(x, y)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 61, in check_flags
    assert x.flags["C_CONTIGUOUS"] is True, (x.flags, y.flags)
AssertionError: ( C_CONTIGUOUS : False
 F_CONTIGUOUS : False
 OWNDATA : True
 WRITEABLE : True
 ALIGNED : True
 UPDATEIFCOPY : False,   C_CONTIGUOUS : True
  F_CONTIGUOUS : False
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  UPDATEIFCOPY : False)

======================================================================
FAIL: pygpu.tests.test_gpu_ndarray.test_mapping_getitem_w_int('uint64', False)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/support.py", line 39, in f
    func(*args, **kwargs)
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_gpu_ndarray.py", line 571, in mapping_getitem_w_int
    _cmp(_a[1:2], a[1:2])
  File "/home/CSEM/mack0242/.pyenv/versions/keras-venv/lib/python2.7/site-packages/pygpu-0.2.1-py2.7-linux-x86_64.egg/pygpu/tests/test_gpu_ndarray.py", line 618, in _cmp
    assert x.strides == y.strides
AssertionError
----------------------------------------------------------------------
Ran 6904 tests in 55.147s

FAILED (errors=302, failures=569)
abergeron commented 8 years ago

For most of these errors it seems like numpy is using nonsensical values. I would not consider this a pygpu bug but rather a numpy one. We do have versions of numpy compiled with MKL that work so something is wrong with yours.

theano-bot commented 8 years ago

Did you compile numpy from the dev version and not a release version? If so, the put non sensible value for stride in some cases.

Le 17 oct. 2016 10:40, "abergeron" notifications@github.com a écrit :

For most of these errors it seems like numpy is using nonsensical values. I would not consider this a pygpu bug but rather a numpy one. We do have versions of numpy compiled with MKL that work so something is wrong with yours.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Theano/libgpuarray/issues/247#issuecomment-254226835, or mute the thread https://github.com/notifications/unsubscribe-auth/ATYWc2IoB-CkbmjTLVhF5uY5b403L3dPks5q04jGgaJpZM4J0rSC .[image: Web Bug from https://github.com/notifications/beacon/ATYWcwEaQ_0IjrR8qb-2FjtPmpVU-Nl7ks5q04jGgaJpZM4J0rSC.gif] {"api_version":"1.0","publisher":{"api_key":" 05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity": {"external_key":"github/Theano/libgpuarray","title":" Theano/libgpuarray","subtitle":"GitHub repository","main_image_url":" https://cloud.githubusercontent.com/assets/143418/17495839/a5054eac-5d88- 11e6-95fc-7290892c7bb5.png","avatar_image_url":"https:// cloud.githubusercontent.com/assets/143418/15842166/ 7c72db34-2c0b-11e6-9aed-b52498112777.png","action":{"name":"Open in GitHub","url":"https://github.com/Theano/libgpuarray"}}," updates":{"snippets":[{"icon":"PERSON","message":"@abergeron in #247: For most of these errors it seems like numpy is using nonsensical values. I would not consider this a pygpu bug but rather a numpy one. We do have versions of numpy compiled with MKL that work so something is wrong with yours."}],"action":{"name":"View Issue","url":"https://github. com/Theano/libgpuarray/issues/247#issuecomment-254226835"}}}

abergeron commented 7 years ago

The new test failures are indeed from the special stride values that numpy is using in the development versions.

The one error that worries me is the unaligned array one in pygpu.tests.test_tools.test_check_args_collapse_2, but I can't manage to reproduce it.