molpopgen / fwdpp

fwdpp is a C++ template library for implementing efficient forward-time population genetic simulations
http://fwdpp.readthedocs.io
GNU General Public License v3.0
27 stars 11 forks source link

Add ability to serialize population containers in chunks. #92

Closed molpopgen closed 6 years ago

molpopgen commented 6 years ago

One problem with the current serialization methods is that it is possible to run out of RAM when serializing an entire population into a buffer. For some applications, like writing to a compressed files, you don't need the entire pop in memory in binary format. Rather, you could get the data into a buffer in chunks and then process the buffer.

This should be possible by adding onto the changes in #90.

molpopgen commented 6 years ago

Closing. In hindsight, #90 deals with this directly. The main challenge in practice is the availability of fully-featured compressed stream libraries.