Open carlodefalco opened 1 year ago
as a temporary workaround suggest using compressed output via boost::iostreams, an example is provided with this cset : ed0e798
as a reference, on my system I get :
-rw-r--r-- 1 carlo staff 26K 29 Mag 09:35 esempio.json.bz
-rw-r--r-- 1 carlo staff 33K 29 Mag 09:39 esempio.json.gz
-rw-r--r-- 1 carlo staff 243K 29 Mag 09:39 esempio2.json
which means a deflation factor of 7.4 with gzip and 9.3 with bzip (which is supposedly slower)
for compilation boost headers and libraries need to be added
mpicxx -std=c++17 ... -lboost_iostreams-mt -o esempio_json_gz esempio_json_gz.cpp ../src/particles.cpp
added also an example for csv/octave_ascii output with commit 2ec4465
-rw-r--r-- 1 carlo staff 107K 29 Mag 14:43 esempio.csv
-rw-r--r-- 1 carlo staff 32K 29 Mag 14:43 esempio.csv.gz
-rw-r--r-- 1 carlo staff 30K 29 Mag 14:52 esempio.octtxt.gz
-rw-r--r-- 1 carlo staff 100K 29 Mag 14:52 esempio.octtxt
the deflation factor seems to be around 3.3
Particle data files tend to be of VERY large size, so a binary (and possibly compressed) data file format would be really helpful ...
Consider output in, e.g. : https://www.hdfgroup.org/solutions/hdf5/ or possibly use some library for serialization of STL containers such as : https://uscilab.github.io/cereal/stl_support.html