As a quick reminder, openPMD is not a file format or even a format per se, it's a meta-data markup. We just implement it on top of popular hierarchical file formats as a reference.
We should consider several updates to openPMD to make streaming workflows more natural.
IterationEncoding in Series
A third iteration encoding mode in data series, maybe called "variableBased" could express that new iterations (snapshots) of data are stored as as implementation-specific "variable update" of the data. This is what ADIOS supports for example (begin and end step on a variable). Other kinds of storage media, such as object storage and databases of all kinds implement similar things.
Attributes
On a high level, if an attribute did not change since the last iteration (snapshot), it might be allowed to be omitted. Potentially we want to add a reference mechanism to "link" the last iteration (snapshot) that had the meta-data attributes of interest.
As a quick reminder, openPMD is not a file format or even a format per se, it's a meta-data markup. We just implement it on top of popular hierarchical file formats as a reference.
@franzpoeschel et al. already implemented working implementations of streaming openPMD data and meta-data with ADIOS2 SST https://github.com/openPMD/openPMD-api/pull/570.
We should consider several updates to openPMD to make streaming workflows more natural.
IterationEncoding in Series
A third iteration encoding mode in data series, maybe called "variableBased" could express that new iterations (snapshots) of data are stored as as implementation-specific "variable update" of the data. This is what ADIOS supports for example (begin and end step on a variable). Other kinds of storage media, such as object storage and databases of all kinds implement similar things.
Attributes
On a high level, if an attribute did not change since the last iteration (snapshot), it might be allowed to be omitted. Potentially we want to add a reference mechanism to "link" the last iteration (snapshot) that had the meta-data attributes of interest.