search

ali-ramadhan / lagrangian-microbes

🦐🐟🦈 A framework for simulating millions of interacting Lagrangian particles (or microbes!) in a turbulent ocean.
MIT License
10 stars 0 forks source link

Better way of writing output? #7

Closed ali-ramadhan closed 5 years ago

ali-ramadhan commented 5 years ago

Right now everything is written to uncompressed pickle files. At the very least, the pickle files should be compressed. Would be good to use NetCDF as it's a familiar format.

Unfortunately last time I tried, Parcels' NetCDF output didn't work when running parallel jobs with joblib so I had to use joblib pickles. If this is still the case, maybe all the pickle files can be merged into a massive NetCDF file at the end?

The interactions might create Lagrangian microbes with properties such as species (integer or string) or genome (binary string). These should still fit in NetCDF files.

ali-ramadhan commented 5 years ago

Zarr might be interesting to look at.

ali-ramadhan commented 5 years ago

PR #24 resolves this issue.