Open cosmonaut-ok opened 3 years ago
Hi @cosmonaut-ok,
Thank you for the idea!
We plan something very similar, currently drafted as "variableBased" encoding related to #221. In that, we will use the intrinsic capabilities of e.g. the ADIOS2 format to encode steps inside a data set (variable) if the file format supports this.
n+1
dimensionality in other formats, e.g. HDF5, is thinkable as a work-around for those formats. Yet we need to be aware that contrary to ADIOS1/2 "step" encoding, HDF5 would then be limited to stay with the same shape of the data set for all iterations; which is limiting for some applications (e.g. particle data shape changes over time).
It's not comfortable to make separate file/group for each iteration in case of active working with time series (e.g. if required to select single point from every iteration). Much better to make
n+1
-dimensional array and write data to it. But, it breaks openPMD compatibility.