ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

GPU Unpack, take four #107

Closed jaycedowell closed 6 years ago

jaycedowell commented 6 years ago

Add support for GPU-based unpacking and quantization (fourth attempt). This includes:

codecov-io commented 6 years ago

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@e1140af). Click here to learn what that means. The diff coverage is 75%.

Impacted file tree graph

@@           Coverage Diff            @@
##             master    #107   +/-   ##
========================================
  Coverage          ?   58.8%           
========================================
  Files             ?      63           
  Lines             ?    5093           
  Branches          ?       0           
========================================
  Hits              ?    2995           
  Misses            ?    2098           
  Partials          ?       0
Impacted Files Coverage Δ
python/bifrost/block.py 89.79% <100%> (ø)
python/bifrost/DataType.py 63.49% <66.66%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update e1140af...e2ee6d5. Read the comment docs.

jaycedowell commented 6 years ago

This is like take three but puts the new GPU quantizer/unpacker into .cu and .hu files to preserve the history in quantize.cpp and unpack.cpp. This also adds additional unit tests for the GPU functions and the promotion functions.

coveralls commented 6 years ago

Coverage Status

Coverage increased (+0.006%) to 62.201% when pulling 75d7896e4ad78e8e367492a79a1723d8e0d33b64 on jaycedowell-gpu-unpack2 into cd5161a02ae77ee6e23c2c8777ed254c9316b08a on master.

MilesCranmer commented 6 years ago

Looks good to me! @benbarsdell do you have any issues with this?

benbarsdell commented 6 years ago

I'm not sure what happened to the CI tests here; any ideas?. All tests pass when I run this branch locally.

jaycedowell commented 6 years ago

I'm not sure what is going on here. I just triggered a build of master and ended up with the same error that kills this PR. I think the culprit may be:

test_block.py:315: UserWarning: loadtxt: Empty input file: ".log.txt" result = np.loadtxt(self.logfile).astype(np.float32).view(np.complex64)

although I don't fully understand what this means.

jaycedowell commented 6 years ago

That's interesting. I restarted a build from the jaycedowell-gpu-unpack2 branch that was previously marked as successful and it failed.

jaycedowell commented 6 years ago

Ah, it's the NumPy version. I rolled back the NumPy version in the .travis.yaml file from the latest (1.14.3) to 1.13.3 using the jaycedowell-travistest branch and the build works.

jaycedowell commented 6 years ago

Making sure that files are closed after np.savetxt() is called with a file handle fixes the testing problem.

benbarsdell commented 6 years ago

Thanks for tracking that down Jayce!