carlodefalco / quadgrid

simple cartesian quad grid for c++/octave
https://carlodefalco.github.io/quadgrid/
BSD 2-Clause "Simplified" License
2 stars 1 forks source link

Implement output in binary format #6

Open carlodefalco opened 1 year ago

carlodefalco commented 1 year ago

Particle data files tend to be of VERY large size, so a binary (and possibly compressed) data file format would be really helpful ...

Consider output in, e.g. : https://www.hdfgroup.org/solutions/hdf5/ or possibly use some library for serialization of STL containers such as : https://uscilab.github.io/cereal/stl_support.html

carlodefalco commented 1 year ago

as a temporary workaround suggest using compressed output via boost::iostreams, an example is provided with this cset : ed0e798

as a reference, on my system I get :

-rw-r--r--   1 carlo  staff    26K 29 Mag 09:35 esempio.json.bz
-rw-r--r--   1 carlo  staff    33K 29 Mag 09:39 esempio.json.gz
-rw-r--r--   1 carlo  staff   243K 29 Mag 09:39 esempio2.json

which means a deflation factor of 7.4 with gzip and 9.3 with bzip (which is supposedly slower)

for compilation boost headers and libraries need to be added

mpicxx -std=c++17 ...  -lboost_iostreams-mt -o esempio_json_gz esempio_json_gz.cpp ../src/particles.cpp 
carlodefalco commented 1 year ago

added also an example for csv/octave_ascii output with commit 2ec4465

-rw-r--r--   1 carlo  staff   107K 29 Mag 14:43 esempio.csv
-rw-r--r--   1 carlo  staff    32K 29 Mag 14:43 esempio.csv.gz
-rw-r--r--   1 carlo  staff    30K 29 Mag 14:52 esempio.octtxt.gz
-rw-r--r--   1 carlo  staff   100K 29 Mag 14:52 esempio.octtxt

the deflation factor seems to be around 3.3