Sophisticated analyses use high-dimensional histograms, both disc and memory usage can become an issue. Disc usage is easy to fix by writing a compressed stream with Boost.Serialization and Boost.IOStreams. But this does not fix the memory issue.
It would be interesting to add an experimental memory-compressed storage and see how that performs. In-memory compression can be achieved in several ways.
Zero-suppression: Some high-dimensional histograms contain a lot of zeros. A simple compression would just replace all the zeros with a code that indicates how many zeros are in the omitted gap. This is comparably easy to implement. Frequent re-allocations of memory in the beginning will be an issue, but that could be optimized over time.
Sophisticated analyses use high-dimensional histograms, both disc and memory usage can become an issue. Disc usage is easy to fix by writing a compressed stream with Boost.Serialization and Boost.IOStreams. But this does not fix the memory issue.
It would be interesting to add an experimental memory-compressed storage and see how that performs. In-memory compression can be achieved in several ways.