Merge initial round of benchmarks

[MilesCranmer]

• Create benchmarks modules (optional install inside test/benchmarks)
  • Adds PipelineBenchmarker class, which is basically an entire class built around @telegraphic's timeit function, which lets you record timeit's for various specific parts of your pipeline and its blocks
  • (Can probably do some of this with /dev/shm logging and Jayce's utility's)
• Log of historical benchmark scores in .csv. This will probably be machine-generated later by a CI service, but I have put an example in there right now
• Benchmark for compile time of Bifrost
• Benchmark for read_sigproc block
• WIP Benchmark for a GPU FFT pipeline
• Syntax benchmark comparing Danny's gpuspec code with Bifrost against the legacy, pure-C++ code
  • test/benchmarks/development_vs_gpuspec/Makefile
  • Run with make test, the output is:

    With Bifrost:
    ==========================================================================================
    Total nloc   Avg.NLOC  AvgCCN  Avg.token   Fun Cnt  Warning cnt   Fun Rt   nloc Rt
    ------------------------------------------------------------------------------------------
    165       6.1     1.2       51.9       16            0      0.00    0.00
    ==========================================================================================
    Without Bifrost:
    ==========================================================================================
    Total nloc   Avg.NLOC  AvgCCN  Avg.token   Fun Cnt  Warning cnt   Fun Rt   nloc Rt
    ------------------------------------------------------------------------------------------
    1329      23.0     5.2      247.8       51            1      0.02    0.41
    ==========================================================================================

I do some preprocessing on Danny's code to remove the extra (?) Hdf block, and only select files in both repo's that I think are relevant to the calculation. See WITH_BIFROST and WITHOUT_BIFROST for the list of files I include. The above numbers are important to get right as they are the only benchmarks scores (?) to be listed in the paper other than packet capture rate/other LWA Sev pipelines. This is pending further discussion, but that is what I recall we had decided.

Output via lizard

[MilesCranmer]

New output, taking into account Danny's comments during latest telecon (on which files to use). I also do zero preprocessing now, because it doesn't look like there's an extra Hdf* block.

With Bifrost:
==========================================================================================
Total nloc   Avg.NLOC  AvgCCN  Avg.token   Fun Cnt  Warning cnt   Fun Rt   nloc Rt
------------------------------------------------------------------------------------------
       177       5.5     1.2       49.5        8            0      0.00    0.00
==========================================================================================
Without Bifrost:
==========================================================================================
Total nloc   Avg.NLOC  AvgCCN  Avg.token   Fun Cnt  Warning cnt   Fun Rt   nloc Rt
------------------------------------------------------------------------------------------
      1329      23.0     5.2      247.8       51            1      0.02    0.41
==========================================================================================

This is the list of files currently used:

• Bifrost (telegraphic/bunyip): blocks/accumulate.py blocks/hdf.py bf_gpuspec.py
• Non-Bifrost (UCBerkeleySETI/gbt_seti): guppi2spectra.c guppi2spectra_gpu.cu guppi2spectra_gpu.h filterbank.h

@benbarsdell, let me know if you think this is merge-able. The work is all inside a new folder: test/benchmarks, so there should not be any conflicts with the main Bifrost code

[coveralls]

Coverage decreased (-0.007%) to 65.194% when pulling ec97c5d52887eb30b764c0b1188ee6dbf56c1bd2 on benchmarks into 8a4a17d707619db771a9e970b408c72f8072dd71 on master.

[codecov-io]

Codecov Report

Merging #96 into master will increase coverage by 0.29%. The diff coverage is 73.98%.

@@            Coverage Diff             @@
##           master      #96      +/-   ##
==========================================
+ Coverage   69.26%   69.55%   +0.29%     
==========================================
  Files          34       34              
  Lines        3377     3426      +49     
==========================================
+ Hits         2339     2383      +44     
- Misses       1038     1043       +5

Impacted Files	Coverage Δ
python/bifrost/blocks/binary_io.py	100% <ø> (ø)	:arrow_up:
python/bifrost/ring.py	92.74% <100%> (+0.07%)	:arrow_up:
python/bifrost/device.py	63.63% <100%> (ø)	:arrow_up:
python/bifrost/Space.py	84.21% <100%> (+0.87%)	:arrow_up:
python/bifrost/pipeline.py	84.32% <50%> (ø)	:arrow_up:
python/bifrost/dtype.py	34.28% <50%> (+12.69%)	:arrow_up:
python/bifrost/libbifrost.py	68.1% <70.96%> (+1.1%)	:arrow_up:
python/bifrost/DataType.py	65.71% <75%> (+2.07%)	:arrow_up:
python/bifrost/ndarray.py	77.57% <80%> (+0.1%)	:arrow_up:
python/bifrost/ring2.py	88.53% <96.15%> (-0.11%)	:arrow_down:

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8a4a17d...bb4272e. Read the comment docs.

[MilesCranmer]

This PR now also includes a variety of speed ups to the Bifrost python frontend, which were shown by the profiler to be major hotspots. Right now these mostly are cases where a PyCLibrary constant was being generated at each hit, rather than being cached by Python. The most expensive cases of this are now globals in their respective files.

[coveralls]

Coverage increased (+0.3%) to 65.489% when pulling bb4272ea56a0251f49bdda0f20a3002fbb28a6e0 on benchmarks into 8a4a17d707619db771a9e970b408c72f8072dd71 on master.

[MilesCranmer]

Latest commits introduce many more performance improvements, as well as benchmark numbers for the linear FFT pipeline courtesy of Ben.

The performance improvements include a commit be373ec which takes all pyclibrary calls in the main Bifrost API and replaces them with a global variable. The performance improvement from this and the few instances where error checking is put into __debug__ statements is quite significant. You need to install with python setup.py install -O2, and run with python -OO ... to see some of this speedup.

After benchmarking against skcuda for an FFT->IFFT pipeline using the GPU, here's the performance figure that's going in the paper:

.

Also, if possible, @benbarsdell, we should merge this soon, and then up the version of Bifrost and make a release so we can reference the version number in the benchmarks described in the paper.