search

blitzpp / blitz

Blitz++ Multi-Dimensional Array Library for C++
https://github.com/blitzpp/blitz/wiki
Other
406 stars 84 forks source link

Build efficient serialization to persist / cache / share Blitz Array #1

Open citibeth opened 8 years ago

citibeth commented 8 years ago

Thomas Vincent Wrote:

One thing that we had to develop (only for 3d array) and I do thing should be part of blitz++ itself is efficient file serialization to persist / cache / share blitz array.

I have stuff that does that in NetCDF, and the idea is to include those subroutines in the Blitz++ library. Is this what you are talking about, or are you think of something different?

No not really, I found volume dump (read/write) really useful when working with big (huge) volume. So we implemented a generic binary read/write based on simple std::fstream::write() / read() to share volume across processes / cache to disk them temporally. Format is potentially not cross platform at all but it's really convenient. We did it for 2 / 3 dimensional array of contiguous data but it should scale nicely to higher rank ... not sure how to approach the non-contiguous part though. I think it's something blitz should offer.

After that being able to read/write specific format is necessary too to spread blitz++ usage I think. I have vtk / paraview structured data skills and code (write part only) if it could help.

maddanio commented 8 years ago

Non contiguous should be ok, as long as you can generate an element wise iterator. also when reading you may want control over contiguity/alignment...

slayoo commented 8 years ago

Not sure how relevant it is, but when you do

#define BZ_HAVE_BOOST_SERIALIZATION
#include <blitz/array.h>

it comes with Boost serialization support out of the box