search

openPMD / openPMD-standard

:notebook: Open Standard for Particle-Mesh Data
http://www.openPMD.org
Creative Commons Attribution 4.0 International
78 stars 26 forks source link

Proposed: make option "recordBased" for iterationEncoding #236

Open cosmonaut-ok opened 3 years ago

cosmonaut-ok commented 3 years ago

It's not comfortable to make separate file/group for each iteration in case of active working with time series (e.g. if required to select single point from every iteration). Much better to make n+1-dimensional array and write data to it. But, it breaks openPMD compatibility.

ax3l commented 3 years ago

Hi @cosmonaut-ok,

Thank you for the idea!

We plan something very similar, currently drafted as "variableBased" encoding related to #221. In that, we will use the intrinsic capabilities of e.g. the ADIOS2 format to encode steps inside a data set (variable) if the file format supports this.

n+1 dimensionality in other formats, e.g. HDF5, is thinkable as a work-around for those formats. Yet we need to be aware that contrary to ADIOS1/2 "step" encoding, HDF5 would then be limited to stay with the same shape of the data set for all iterations; which is limiting for some applications (e.g. particle data shape changes over time).