search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

Jenkins issues #78

Closed MilesCranmer closed 7 years ago

MilesCranmer commented 7 years ago

Jenkins is running again. I'd like to have it output some useful statistics, e.g., on pipeline performance tests, eventually.

Right now, however, it's set up to just run all the unit tests. I am seeing the following issues in test_fft.TestFFT. All other tests pass.

fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:290 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:290 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:290 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
Efft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
fft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
Efft.cu:282 _cufftGetErrorString(cufft_ret) = CUFFT_EXEC_FAILED
fft.cu:282 Condition failed: cufft_ret == CUFFT_SUCCESS
fft.cu:282 error 32: BF_STATUS_DEVICE_ERROR
E....
======================================================================
ERROR: test_pipeline (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_pipeline
Traceback (most recent call last):
  File "/usr/lib/python2.7/unittest/loader.py", line 254, in _find_tests
    module = self._get_module_from_name(name)
  File "/usr/lib/python2.7/unittest/loader.py", line 232, in _get_module_from_name
    __import__(name)
  File "/bifrost/test/test_pipeline.py", line 34, in <module>
    from bifrost.sigproc_block   import read_sigproc
ImportError: No module named sigproc_block

======================================================================
ERROR: test_c2r_1D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 154, in test_c2r_1D
    self.run_test_c2r(self.shape1D, [0])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 156, in test_c2r_2D
    self.run_test_c2r(self.shape2D, [0,1])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 181, in test_c2r_2D_in_4D_dims01
    self.run_test_c2r(self.shape4D, [0,1])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 183, in test_c2r_2D_in_4D_dims02
    self.run_test_c2r(self.shape4D, [0,2])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims03 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 185, in test_c2r_2D_in_4D_dims03
    self.run_test_c2r(self.shape4D, [0,3])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 187, in test_c2r_2D_in_4D_dims12
    self.run_test_c2r(self.shape4D, [1,2])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims13 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 189, in test_c2r_2D_in_4D_dims13
    self.run_test_c2r(self.shape4D, [1,3])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_2D_in_4D_dims23 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 191, in test_c2r_2D_in_4D_dims23
    self.run_test_c2r(self.shape4D, [2,3])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_c2r_3D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 158, in test_c2r_3D
    self.run_test_c2r(self.shape3D, [0,1,2])
  File "/bifrost/test/test_fft.py", line 92, in run_test_c2r
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_1D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 147, in test_r2c_1D
    self.run_test_r2c(self.shape1D, [0])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 149, in test_r2c_2D
    self.run_test_r2c(self.shape2D, [0,1])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 161, in test_r2c_2D_in_3D_dims01
    self.run_test_r2c(self.shape3D, [0,1])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 163, in test_r2c_2D_in_3D_dims02
    self.run_test_r2c(self.shape3D, [0,2])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_3D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 165, in test_r2c_2D_in_3D_dims12
    self.run_test_r2c(self.shape3D, [1,2])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims01 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 168, in test_r2c_2D_in_4D_dims01
    self.run_test_r2c(self.shape4D, [0,1])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims02 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 170, in test_r2c_2D_in_4D_dims02
    self.run_test_r2c(self.shape4D, [0,2])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims03 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 172, in test_r2c_2D_in_4D_dims03
    self.run_test_r2c(self.shape4D, [0,3])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims12 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 174, in test_r2c_2D_in_4D_dims12
    self.run_test_r2c(self.shape4D, [1,2])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims13 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 176, in test_r2c_2D_in_4D_dims13
    self.run_test_r2c(self.shape4D, [1,3])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_2D_in_4D_dims23 (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 178, in test_r2c_2D_in_4D_dims23
    self.run_test_r2c(self.shape4D, [2,3])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR

======================================================================
ERROR: test_r2c_3D (test_fft.TestFFT)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/bifrost/test/test_fft.py", line 151, in test_r2c_3D
    self.run_test_r2c(self.shape3D, [0,1,2])
  File "/bifrost/test/test_fft.py", line 80, in run_test_r2c
    fft.execute(idata, odata)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 54, in execute
    inverse=inverse)
  File "build/bdist.linux-x86_64/egg/bifrost/fft.py", line 61, in execute_workspace
    workspace_ptr, workspace_size) )
  File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 152, in _check
    raise RuntimeError(status_str)
RuntimeError: BF_STATUS_DEVICE_ERROR
benbarsdell commented 7 years ago

What GPU and CUDA version is Jenkins running with again?

benbarsdell commented 7 years ago

Ping. Also, how/where can we see the status of the Jenkins builds?

MilesCranmer commented 7 years ago

Sorry @benbarsdell, Jenkins is on my home computer in Canada which has been shut down for the summer... It's a GTX 980. I think I had installed CUDA 7.5 or 8.0.

We could potentially set up a AWS instance to open/close once in a while and run tests, or once we get ledagpu1/2 running, we could test it on there (though the Harvard network security is intense, would be tricky to post a status anywhere unless we set up some digital Rube Goldberg machine to hop through 50 ssh tunnels)