Compressed HDF5 output

[frankier]

Outputs are currently uncompressed. There should be options for:

• Lossless compression using blosc. There should be some kind of reasonable default which gives an acceptable speed/size tradeoff.
• Lossy compression. e.g. coordinates could be 4.12 fixed point numbers & confidence scores could be 0.8 fixed point numbers. This would half storage off the bat.
  • Could also support float16, but this doesn't really give enough precision for coordinates

[frankier]

This should also be benchmarked by read/write speed on normal and networked storage and compression ratio.

[frankier]

The effect of chunking is also a big issue https://www.unidata.ucar.edu/blogs/developer/entry/chunking_data_why_it_matters

[frankier]

http://alimanfoo.github.io/2016/09/21/genotype-compression-benchmark.html