Writing and Reading Non-openPMD Infos

[ax3l]

It would be necessary to allow storing non-openPMD information as well.

In the simple case, this means adding additional, user-defined attributes to records, record-components and paths.

In the more complex case, application specific states such as RNG states, unqiue-number-generator states, etc. need to be stored as records outside of the openPMD-interpreted paths.

[C0nsultant]

The simple case is already possible, maybe not made explicit enough yet. Every "position" in an openPMD hierearchy (which in this APIs terms is an Attributable) supports a templated operation setAttribute, taking a string and a generic argument whose type has to map to a supported Datatype

Here's an example demonstrating said feature by annotating the root group with a std::string.

[ax3l]

That's already a great start, thx! :)

The latter case is quite relevant for PIConGPU checkpoints.

[ax3l]

cc @ejcjason you also need this, right?

[JunCEEE]

cc @ejcjason you also need this, right?

I cannot access the link "here", but I guess the suggested solution is to change the attribute of "position" under one particle group into something we want. That's good. But what if we want to add one group, say "observable", in the iteration group (i.e., in the same level of particles)?

• data/
  • 1/
    • observable/
    • particles/
      • Cu

It seems that the above solution doesn't work in this case, because there is no command like i.meshes[xx] or i.particles[xx] to create the observable. Maybe it can be achieved if we have something to change the group name of particles or meshes?

[ax3l]

Links updated with permanently accessible examples. setAttribute examples is now also in the manual.

Yes, what you want is to side-channel to the low-level write API to write groups and datasets outside of openPMD. That's also what I meant.

Maybe it can be achieved if we have something to change the group name of particles or meshes?

Or are you already good if you can change the meshesPath and particlesPath that we pick before we write? Probably not really.

[pgrete]

What the current status of supporting reading/writing other group data, @ax3l ? I didn't find any more recent and we have a use case to store data that is not tied to a mesh (or particle).

[franzpoeschel]

There is an openPMD-standard PR here https://github.com/openPMD/openPMD-standard/pull/282 and an openPMD-api PR here https://github.com/openPMD/openPMD-api/pull/1432

The openPMD-api PR can already be used, but there is a number of things that are still up to discussion. If you want to try it out, you can have a look at the Python example or the C++ tests in the Diff of that PR.

[pgrete]

Thanks for the pointer. What's the timescale for those changes? I see that the api PR is open for about a year and I'm hesitant to build our future production backend on changes that are potentially still in flux/subject to discussion.

[franzpoeschel]

What's the timescale for those changes?

This is part of a delivery scheduled for this autumn, but I hope to merge the openPMD-api pull request sooner than that. It is a relatively big change and the API might still be changed / is subject to discussion, so long story short is that there is no stable support for this yet, but there is a timeline.

There are logically two parts to this project:

1. Writing custom datasets into custom hierarchies. This does not affect the openPMD-standard.
2. Using openPMD markup within these custom hierarchies. This affects the openPMD-standard.

The workaround for now would probably be to try if you can "pretend" your custom data is a mesh (we do something similar for checkpointing data in PIConGPU at the moment).