Do we need a codec?

[caseyjlaw]

Data volumes and rates are large and may be limiting either in the correlator or within the realfast cluster. Using compression/decompression may help trade compute for data rate limitations. Prior art:

• https://github.com/aroffringa/dysco
  • Custom built for CASA in C, so hard to port.
  • Algorithm is well suited to low-SNR compression. Not tested yet.
• https://github.com/kiyo-masui/bitshuffle
  • Well designed and portable, esp if using hdf5
  • Requires high correlation between neighboring bytes (i.e., high SNR). Tests so far suggest this is not true for fast dump VLA.

[caseyjlaw]

Tests suggest we are not limited by this issue and that we need to reduce the number of development priorities. Closing.