search

openPMD / openPMD-standard

:notebook: Open Standard for Particle-Mesh Data
http://www.openPMD.org
Creative Commons Attribution 4.0 International
78 stars 27 forks source link

Simplifying iterationEncoding and iterationFormat #166

Open DavidSagan opened 6 years ago

DavidSagan commented 6 years ago

Proposal: Simplify the standard by combining iterationEncoding and iterationFormat into one attribute iterationGrouping (which should be snapshotGrouping if iteration is changed to snapshot (See Issue https://github.com/openPMD/openPMD-standard/issues/148)).

For multiple file based grouping iterationGrouping would look something like: file://filename_%T.h5 And the file:// prefix would indicate that the grouping is file based. and filename_%T.h5 is the pattern used for file matching.

For single file based grouping iterationGrouping would be set to group. One variant of this proposal would be to make iterationGrouping optional with group the default value.

ax3l commented 6 years ago

Thank you for the proposal!

The proposal would reduce the attributes from two to one by moving the logic from the attributes to string parsing. If we agree that string parsing might be simpler than reacting on a single-purpose attribute (this is a pure design decision) we could also go one step further:

We would also have to add a note to implementors on the attribute that file:// is relative to the currently open "file", not to the executable that opens it (can be confusing from an app perspective).

What do you think? Update: in the last VC we agreed this looks feasible and could simplify attributes.

Linked issues: #121

ax3l commented 6 years ago

ccing @C0nsultant

ax3l commented 6 years ago

cc @mccoys

ax3l commented 6 years ago

Note: we should check if / is allowed "alone" or if this would collide in groups or logic.

DavidSagan commented 6 years ago

Thinking about it, since files are relative to the currently open file, I would use something like file:: as the prefix instead if file:// as less confusing.

RemiLehe commented 6 years ago

Agree with @DavidSagan's suggestion. @DavidSagan : Do you want to do a corresponding PR?

ax3l commented 6 years ago

Good point, but let's not reinvent the wheel here :-)

We would use file if present as well-defined as in URIs, where it denotes the scheme (protocol). The following characters decide if it's relative or absolute, depending if they start with a / or not.

An absolute path just starts with file:/// https://en.wikipedia.org/wiki/File_URI_scheme

Nevertheless, I would encourage writing relative paths by default since it makes data series more portable.

DavidSagan commented 6 years ago

@ax3l

An absolute path just starts with file:/// https://en.wikipedia.org/wiki/File_URI_scheme

This might be a bit confusing since in the URI scheme something like file://abc/etc... means that abc is the host name which is not how it should be interpreted. So how about just removing the // so that something likefile:abc/etc is a relative path and file:/abc/etc is an absolute path?

ax3l commented 6 years ago

that's a good point with the hostname in file URIs!

Yes, either without the // (some) or skipping the file:// protocol completely (e.g. curl) seem to be common ways to express relative files. Will research it a bit more. I think I prefer the latter already for relative files.