search

ledatelescope / bifrost

A stream processing framework for high-throughput applications.
BSD 3-Clause "New" or "Revised" License
66 stars 29 forks source link

Python 3 Support and Docker Update #136

Closed jaycedowell closed 4 years ago

jaycedowell commented 4 years ago

This PR adds support for Python 3 through a combination of changes to Bifrost itself as well as a new version of ctypesgen (which also addresses #115). It also updates the Docker GPU image to target Ubuntu 18/CUDA 10.2 (to address #129) and adds the tools in tools to the Python installation.

coveralls commented 4 years ago

Coverage Status

Coverage increased (+0.04%) to 61.701% when pulling 36b42a4e86eae867be35814f7229cb8494f72700 on docker-gpu-update into 5cdaa03555cbf97e2b8205c27935de578feca3b2 on master.

codecov-io commented 4 years ago

Codecov Report

Merging #136 into master will decrease coverage by 0.17%. The diff coverage is 0%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
- Coverage   58.91%   58.73%   -0.18%     
==========================================
  Files          63       64       +1     
  Lines        5140     5155      +15     
==========================================
  Hits         3028     3028              
- Misses       2112     2127      +15
Impacted Files Coverage Δ
python/bifrost/romein.py 0% <0%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update fd754e6...602908f. Read the comment docs.

codecov-io commented 4 years ago

Codecov Report

Merging #136 into master will increase coverage by 0.44%. The diff coverage is 62.35%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
+ Coverage   58.53%   58.97%   +0.44%     
==========================================
  Files          65       64       -1     
  Lines        5117     5155      +38     
==========================================
+ Hits         2995     3040      +45     
+ Misses       2122     2115       -7
Impacted Files Coverage Δ
python/bifrost/__init__.py 80.76% <25.00%> (+0.76%) :arrow_up:
python/bifrost/portaudio.py 4.02% <25.00%> (+0.64%) :arrow_up:
python/bifrost/sigproc.py 87.81% <25.00%> (+0.05%) :arrow_up:
python/bifrost/sigproc2.py 54.24% <33.33%> (+0.21%) :arrow_up:
python/bifrost/blocks/sigproc.py 24.88% <44.44%> (+1.05%) :arrow_up:
python/bifrost/block.py 89.86% <50.00%> (+0.06%) :arrow_up:
python/bifrost/ring2.py 86.70% <50.00%> (+0.04%) :arrow_up:
python/bifrost/blocks/detect.py 22.58% <60.00%> (+3.93%) :arrow_up:
python/bifrost/pipeline.py 84.47% <60.00%> (+0.09%) :arrow_up:
python/bifrost/proclog.py 31.42% <66.66%> (+4.15%) :arrow_up:
... and 10 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update b9140e8...c7a9782. Read the comment docs.

jaycedowell commented 4 years ago

Thanks for taking a look at everything. The "fir" and "memory" changes were in response to some compiler warnings I started seeing under newer versions of Ubuntu/CUDA. It's nothing shocking but I should probably run the test suite on "fir" locally just to make sure it all checks out.

codecov-commenter commented 4 years ago

Codecov Report

Merging #136 into master will increase coverage by 3.01%. The diff coverage is 70.37%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #136      +/-   ##
==========================================
+ Coverage   58.61%   61.62%   +3.01%     
==========================================
  Files          64       64              
  Lines        5110     5300     +190     
==========================================
+ Hits         2995     3266     +271     
+ Misses       2115     2034      -81
Impacted Files Coverage Δ
python/bifrost/GPUArray.py 0.00% <0.00%> (ø)
python/bifrost/blocks/fft.py 16.25% <ø> (ø)
python/bifrost/blocks/reduce.py 23.40% <0.00%> (ø)
python/bifrost/udp_capture.py 0.00% <0.00%> (ø)
python/bifrost/udp_transmit.py 0.00% <0.00%> (ø)
python/bifrost/guppi_raw.py 10.52% <14.28%> (+1.43%) :arrow_up:
python/bifrost/blocks/wav.py 21.92% <23.07%> (+0.36%) :arrow_up:
python/bifrost/portaudio.py 4.02% <25.00%> (+0.64%) :arrow_up:
python/bifrost/psrdada.py 2.70% <25.00%> (-0.03%) :arrow_down:
python/bifrost/sigproc2.py 53.18% <27.27%> (-0.85%) :arrow_down:
... and 51 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 5cdaa03...36b42a4. Read the comment docs.

jaycedowell commented 4 years ago

Ok, I think this is much better now. I've also setup a Jenkins server here at UNM which enables me to run the almost all of the test suite that uses the GPU on both Python 2.7 and Python 3.6. The only thing that I haven't been able to run is the FDMT test suite. I also might look into @telegraphic 's idea of running the scripts under testbench for added confidence.

benbarsdell commented 4 years ago

What's the issue with the FDMT test suite?

jaycedowell commented 4 years ago

When I was initially getting things running on Jenkins I was running out of memory on the FDMT test. Maybe it is worth trying again now that everything else is working.

jaycedowell commented 4 years ago

Huh, the FDMT test seems to work now (and passes). Yay.

jaycedowell commented 4 years ago

@telegraphic I'm having problems running some of the "testbench" examples. test_guppi.py and test_guppi_reader.py don't seem to do anything. test_fft_detect.py initially would not run and still throws errors after I tweaked it.

jaycedowell commented 4 years ago

Does anyone have any final comments on this PR or opinions about updating the Bifrost version to 0.9.0? If so, comment now or forever hold your peace.