Closed MilesCranmer closed 7 years ago
MilesCranmer
commented
7 years ago Could be related: https://bitbucket.org/pypy/pypy/issues/2124/ctypes-from_buffer-c_int-typeerror
They say the most recent version of pypy doesn't have this problem. I'll install from source and check it out.
benbarsdell
commented
7 years ago https://github.com/ledatelescope/bifrost/commit/a0c0376987492af6fb7cce50dfe9644db081e6a3 mostly fixes this. However, the tests that use the old block interface now hang when run with pypy. I haven't looked into why or where they are getting stuck.
(Note that I also tried a nightly build of pypy, but it didn't make any difference).
MilesCranmer
commented
7 years ago Thanks, @benbarsdell !
Unfortunately it's still broken for me. It could be because I'm not using the nightly build. I initially tried building from source but rpython started drawing rainbow mandelbrot fractals for 2 hours as part of its translation process (seriously: http://rpython.readthedocs.io/en/latest/faq.html#why-does-the-translator-draw-a-mandelbrot-fractal-while-translating) and I ended it.
Here's the error I'm getting currently, which could just be a result of a broken build environment:
======================================================================
ERROR: test_ndarray (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: test_ndarray
Traceback (most recent call last):
File "/root/pypy2-v5.8.0-linux64/lib-python/2.7/unittest/loader.py", line 254, in _find_tests
module = self._get_module_from_name(name)
File "/root/pypy2-v5.8.0-linux64/lib-python/2.7/unittest/loader.py", line 232, in _get_module_from_name
__import__(name)
File "/workspace/test/test_ndarray.py", line 31, in <module>
import bifrost as bf
File "build/bdist.linux-x86_64/egg/bifrost/__init__.py", line 50, in <module>
import pipeline
File "build/bdist.linux-x86_64/egg/bifrost/pipeline.py", line 45, in <module>
bf.device.set_devices_no_spin_cpu()
AttributeError: 'module' object has no attribute 'device'Are you testing inside the Bifrost docker, by the way?
benbarsdell
commented
7 years ago That's strange. Do you only get that error with pypy? I only added the bf.device.set_devices_no_spin_cpu() function recently, but bf.device should have been in the namespace for a while.
I haven't tested inside the docker container lately.
MilesCranmer
commented
7 years ago I don't see those specific errors with regular python, but I do see some a few different issues (only for the GPU tests):
MilesCranmer
commented
7 years ago (Sorry about the above error list, it was a result of my stale build!)
BTW, is it OK if I set a variable in the makefiles called PYTHON, which by default is set to python, but which can optionally (manually) be set to, e.g., pypy?
We can also probably get travis to test pypy on separate builds, if we want to "officially" support it. I suppose this is only after we could demonstrate any speedup to motivate that.
These are the only errors still going through (regular python, updated code):
======================================================================
ERROR: test_matmul_aa_ci8 (test_linalg.TestLinAlg)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/test/test_linalg.py", line 166, in test_matmul_aa_ci8
self.run_test_matmul_aa_ci8_transpose(False)
File "/workspace/test/test_linalg.py", line 159, in run_test_matmul_aa_ci8_transpose
self.run_test_matmul_aa_ci8_shape((11,23), transpose=transpose)
File "/workspace/test/test_linalg.py", line 61, in run_test_matmul_aa_ci8_shape
self.linalg.matmul(1, a, None, 0, c)
File "build/bdist.linux-x86_64/egg/bifrost/linalg.py", line 58, in matmul
beta, c_array))
File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 154, in _check
raise RuntimeError(status_str)
RuntimeError: BF_STATUS_UNSUPPORTED_DTYPE
======================================================================
ERROR: test_matmul_ab_ci8 (test_linalg.TestLinAlg)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspace/test/test_linalg.py", line 176, in test_matmul_ab_ci8
self.run_test_matmul_ab_ci8_transpose(False)
File "/workspace/test/test_linalg.py", line 169, in run_test_matmul_ab_ci8_transpose
self.run_test_matmul_ab_ci8_shape((11,23), 7777, transpose=transpose)
File "/workspace/test/test_linalg.py", line 95, in run_test_matmul_ab_ci8_shape
self.linalg.matmul(1, a, b, 0, c)
File "build/bdist.linux-x86_64/egg/bifrost/linalg.py", line 58, in matmul
beta, c_array))
File "build/bdist.linux-x86_64/egg/bifrost/libbifrost.py", line 154, in _check
raise RuntimeError(status_str)
RuntimeError: BF_STATUS_UNSUPPORTED_DTYPE
----------------------------------------------------------------------PYTHON variable sounds good.
PyPy tests with travis would be great if it's easy to set up, but we don't have to officially support it in the short term.
Those errors are probably because you're using an older version of CUDA and/or GPU. Not sure what the best way to solve it is (maybe we can gate the tests on the cuda/gpu version?).
MilesCranmer
commented
7 years ago Thanks. Sorry for my late response.
Update on PyPy: I have added it to travis's build matrix - https://travis-ci.org/ledatelescope/bifrost
PyPy-5.4.1 will now also be tested on Bifrost on each new commit.
This is not a bug, since we've never tested with PyPy. If you want to try pypy, I have instructions at the end
For Bifrost benchmarks, I'm trying out PyPy, since it should give plenty of speedup on any potential python overhead. See http://speed.pypy.org/.
I became very hopeful as I was getting it set up, since everything installed correctly, and I could even import bifrost without errors. However, on trying the test suite, I get a TypeError on almost every run. I think this could be a simple fix, but it could be a mess also.
Here are some of the tests (before stdout was drowned with errors):
EEEEEEEEEEEEEEEEEEEEE........................EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE(So there is maybe one test file that works...)Here is one of the two flavours of errors (is this one just picking up the other error?):
And another (this one makes up the vast majority of errors):
or, e.g.,
It does look like some other tests pass, maybe 2/3 (I see a ..E before each error).
To try out pypy, download a binary, e.g., on ubuntu 14.04: https://bitbucket.org/pypy/pypy/downloads/pypy2-v5.8.0-linux32.tar.bz2. Run the following commands:
Then, in the Bifrost directory,
This will set up pypy with Bifrost. You can launch the interpreter with
pypy, and import bifrost from there.