search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

FFT crashes when using Maxwell card #117

Closed telegraphic closed 2 years ago

telegraphic commented 6 years ago

The bifrost fft is working for me on 1080 cards, but crashes on TitanX (maxwell).

Here's the simplest way for me to recreate:

import numpy as np, bifrost as bf
shape = (128, 128, 256)
known_data = np.random.normal(size=shape).astype(np.float32).view(np.complex64)
idata = bf.ndarray(known_data, space='cuda')
odata = bf.empty_like(idata)
fft = bf.fft.Fft()
fft.init(idata, odata, axes=[0, 1], apply_fftshift=True)

terminate called after throwing an instance of 'thrust::system::system_error'
  what():  function_attributes(): after cudaFuncGetAttributes: invalid device function
Aborted

If I recompile with architectures 30 and 32 (in user.mk), I get a bit further, but it still fails:

fft.execute(idata, odata)
fft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-4-3a46866472e0> in <module>()
----> 1 fft.execute(idata, odata)

/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.pyc in execute(self, iarray, oarray, inverse)
     50         return self.execute_workspace(iarray, oarray,
     51                                       workspace_ptr=None, workspace_size=0,
---> 52                                       inverse=inverse)
     53     def execute_workspace(self, iarray, oarray, workspace_ptr, workspace_size,
     54                           inverse=False):

/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.pyc in execute_workspace(self, iarray, oarray, workspace_ptr, workspace_size, inverse)
     58             asarray(oarray).as_BFarray(),
     59             inverse,
---> 60             workspace_ptr, workspace_size))
     61         return oarray

/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.pyc in _check(status)
    101             else:
    102                 status_str = _bf.bfGetStatusString(status)
--> 103                 raise RuntimeError(status_str)
    104     else:
    105         if status == _bf.BF_STATUS_END_OF_DATA:

RuntimeError: BF_STATUS_DEVICE_ERROR

I'm using CUDA 8.0 on Ubuntu 16.04:

> nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Tue_Jan_10_13:22:03_CST_2017
Cuda compilation tools, release 8.0, V8.0.61

> g++ --version
g++ (Ubuntu 5.4.0-6ubuntu1~16.04.4) 5.4.0 20160609
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

> nvidia-smi
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 375.39                 Driver Version: 375.39                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX TIT...  Off  | 0000:82:00.0     Off |                  N/A |
|  0%   55C    P0    54W / 250W |      0MiB / 12207MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
telegraphic commented 6 years ago

The unit tests for test_fft.py also fail, output below. The test_map unittests pass, so suggests it's FFT related?

(dcpy) dancpr@blc30:~/install/bifrost/test$ python -m unittest test_fft
fft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:307 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:307 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:307 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:323 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:323 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:323 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
Efft.cu:315 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:315 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:315 error 66: BF_STATUS_DEVICE_ERROR
E
======================================================================
ERROR: test_1D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 143, in test_1D
    self.run_test_c2c(self.shape1D, [0])
  File "test_fft.py", line 136, in run_test_c2c
    self.run_test_c2c_impl(shape, axes, fftshift=True)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_1D_in_2D_dim0 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 145, in test_1D_in_2D_dim0
    self.run_test_c2c(self.shape2D, [0])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_1D_in_2D_dim1 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 147, in test_1D_in_2D_dim1
    self.run_test_c2c(self.shape2D, [1])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_1D_in_3D_dim0 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 149, in test_1D_in_3D_dim0
    self.run_test_c2c(self.shape3D, [0])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_1D_in_3D_dim1 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 151, in test_1D_in_3D_dim1
    self.run_test_c2c(self.shape3D, [1])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_1D_in_3D_dim2 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 153, in test_1D_in_3D_dim2
    self.run_test_c2c(self.shape3D, [2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 156, in test_2D
    self.run_test_c2c(self.shape2D, [0, 1])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_3D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 158, in test_2D_in_3D_dims01
    self.run_test_c2c(self.shape3D, [0, 1])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_3D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 160, in test_2D_in_3D_dims02
    self.run_test_c2c(self.shape3D, [0, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_3D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 162, in test_2D_in_3D_dims12
    self.run_test_c2c(self.shape3D, [1, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 164, in test_2D_in_4D_dims01
    self.run_test_c2c(self.shape4D, [0, 1])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 166, in test_2D_in_4D_dims02
    self.run_test_c2c(self.shape4D, [0, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims03 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 169, in test_2D_in_4D_dims03
    self.run_test_c2c(self.shape4D, [0, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 171, in test_2D_in_4D_dims12
    self.run_test_c2c(self.shape4D, [1, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims13 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 173, in test_2D_in_4D_dims13
    self.run_test_c2c(self.shape4D, [1, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_2D_in_4D_dims23 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 175, in test_2D_in_4D_dims23
    self.run_test_c2c(self.shape4D, [2, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_3D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 178, in test_3D
    self.run_test_c2c(self.shape3D, [0, 1, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_3D_in_4D_dims012 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 180, in test_3D_in_4D_dims012
    self.run_test_c2c(self.shape4D, [0, 1, 2])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_3D_in_4D_dims013 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 182, in test_3D_in_4D_dims013
    self.run_test_c2c(self.shape4D, [0, 1, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_3D_in_4D_dims023 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 184, in test_3D_in_4D_dims023
    self.run_test_c2c(self.shape4D, [0, 2, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_3D_in_4D_dims123 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 186, in test_3D_in_4D_dims123
    self.run_test_c2c(self.shape4D, [1, 2, 3])
  File "test_fft.py", line 134, in run_test_c2c
    self.run_test_c2c_impl(shape, axes)
  File "test_fft.py", line 71, in run_test_c2c_impl
    fft.execute(idata, odata, inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_1D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 196, in test_c2r_1D
    self.run_test_c2r(self.shape1D, [0])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 198, in test_c2r_2D
    self.run_test_c2r(self.shape2D, [0, 1])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 224, in test_c2r_2D_in_4D_dims01
    self.run_test_c2r(self.shape4D, [0, 1])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 226, in test_c2r_2D_in_4D_dims02
    self.run_test_c2r(self.shape4D, [0, 2])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims03 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 229, in test_c2r_2D_in_4D_dims03
    self.run_test_c2r(self.shape4D, [0, 3])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 231, in test_c2r_2D_in_4D_dims12
    self.run_test_c2r(self.shape4D, [1, 2])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims13 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 233, in test_c2r_2D_in_4D_dims13
    self.run_test_c2r(self.shape4D, [1, 3])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims23 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 235, in test_c2r_2D_in_4D_dims23
    self.run_test_c2r(self.shape4D, [2, 3])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_3D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 200, in test_c2r_3D
    self.run_test_c2r(self.shape3D, [0, 1, 2])
  File "test_fft.py", line 139, in run_test_c2r
    self.run_test_c2r_impl(shape, axes)
  File "test_fft.py", line 126, in run_test_c2r_impl
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_1D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 189, in test_r2c_1D
    self.run_test_r2c(self.shape1D, [0])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 191, in test_r2c_2D
    self.run_test_r2c(self.shape2D, [0, 1])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 203, in test_r2c_2D_in_3D_dims01
    self.run_test_r2c(self.shape3D, [0, 1])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 205, in test_r2c_2D_in_3D_dims02
    self.run_test_r2c(self.shape3D, [0, 2])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 207, in test_r2c_2D_in_3D_dims12
    self.run_test_r2c(self.shape3D, [1, 2])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 210, in test_r2c_2D_in_4D_dims01
    self.run_test_r2c(self.shape4D, [0, 1])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 212, in test_r2c_2D_in_4D_dims02
    self.run_test_r2c(self.shape4D, [0, 2])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims03 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 215, in test_r2c_2D_in_4D_dims03
    self.run_test_r2c(self.shape4D, [0, 3])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 217, in test_r2c_2D_in_4D_dims12
    self.run_test_r2c(self.shape4D, [1, 2])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims13 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 219, in test_r2c_2D_in_4D_dims13
    self.run_test_r2c(self.shape4D, [1, 3])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims23 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 221, in test_r2c_2D_in_4D_dims23
    self.run_test_r2c(self.shape4D, [2, 3])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_3D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test_fft.py", line 193, in test_r2c_3D
    self.run_test_r2c(self.shape3D, [0, 1, 2])
  File "test_fft.py", line 107, in run_test_r2c
    self.run_test_r2c_dtype(shape, axes, np.float32)
  File "test_fft.py", line 103, in run_test_r2c_dtype
    fft.execute(idata, odata)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 52, in execute
    inverse=inverse)
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/fft.py", line 60, in execute_workspace
    workspace_ptr, workspace_size))
  File "/opt/pyve/dcpy/local/lib/python2.7/site-packages/bifrost-0.8.0-py2.7.egg/bifrost/libbifrost.py", line 103, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

----------------------------------------------------------------------
Ran 42 tests in 23.585s

FAILED (errors=42)
telegraphic commented 6 years ago

further note: tests pass on identical machine with 1080 GPU

benbarsdell commented 6 years ago

Can you try this and let me know if it fixes it:

In src/Makefile, replace: nvprune -o $@ $(NVCC_GENCODE) $(CUFFT_STATIC_GENCODE) $< with: cp $< $@

If it does, I'll try to implement a proper solution.

telegraphic commented 6 years ago

@benbarsdell unfortunately that didn't seem to fix the issue. Is there any further debugging I can do?

jaycedowell commented 6 years ago

Does it work if you re-compile with "GPU_ARCHS ?= 50"?

telegraphic commented 2 years ago

Ugh, ran into this issue again with a recent install, and tracked it down to needing to add the X0 version (GPU_ARCHS = 60 61) so @jaycedowell I think your fix is spot on 🎯

jaycedowell commented 2 years ago

Ok, then we need to update autoconf for #157/#158 so that we keep the X0 arch. as long as the version of CUDA supports it.

jaycedowell commented 2 years ago

Closing with the release of v0.10.0.