OpenPMD-2.0 compliance for the test/example file `bmad_particles.h5`

[s-sajid-ali]

The test/example file bmad_particles.h5 (at openPMD-beamphysics/docs/examples/data/) is currently incompatible with the OpenPMD-1.1 standard as shown by the following errors raised by

openpmd-ls:

sasyed@MAC-140753 ~/D/p/o/d/e/data (master)> openpmd-ls bmad_particles.h5                                                                                       (openpmd-viz) 
[~Series] An error occurred: fileBased output can not be written with no iterations.
An error occurred while opening the specified openPMD series!
Read Error in frontend 
Object type:    Attribute
Error type: UnexpectedContent
Further description:    Unexpected Attribute datatype for 'openPMDextension' (expected uint32, found STRING)
sasyed@MAC-140753 ~/D/p/o/d/e/data (master) [2]>       

openPMD_check_h5:

sasyed@MAC-140753 ~/D/p/o/d/e/data (master)> openPMD_check_h5 -i bmad_particles.h5                                                                              (openpmd-viz) 
Error: Attribute openPMDextension in `/` is not of type 'uint32' (is 'bytes_')!
Error: Attribute iterationEncoding (required) does NOT exist in `/`!
Error: Attribute iterationFormat (required) does NOT exist in `/`!
Warning: Attribute author (recommended) does NOT exist in `/`!
Error: Attribute date in `/` does not satisfy format ('2019-12-02 20:06:54' should be in format '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2} [\+|-][0-9]{4}$')!
Traceback (most recent call last):
  File "/Users/sasyed/Documents/packages/mambaforge/envs/openpmd-viz/bin/openPMD_check_h5", line 10, in <module>
    sys.exit(main())
             ^^^^^^
  File "/Users/sasyed/Documents/packages/mambaforge/envs/openpmd-viz/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 931, in main
    result_array = check_file(file_name, verbose, force_extension_pic)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sasyed/Documents/packages/mambaforge/envs/openpmd-viz/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 917, in check_file
    extensionStates = get_extensions(f, verbose)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/sasyed/Documents/packages/mambaforge/envs/openpmd-viz/lib/python3.12/site-packages/openpmd_validator/check_h5.py", line 116, in get_extensions
    if (bitmask & extensionIDs) == bitmask:
        ~~~~~~~~^~~~~~~~~~~~~~
TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe''
sasyed@MAC-140753 ~/D/p/o/d/e/data (master) [1]> 

Could the example file be updated so as to make it compliant with the base OpenPMD-1.1.0 specification? Thanks!

I am working (alongside @ax3l) to update the visualization tools for OpenPMD artifacts as part of the CAMPA project. We are looking to contribute a simplified backend which uses the Python bindings of openpmd-api instead of h5py to allow for the usage of ADIOS2 files in openpmd-beamphysics once the above issue is fixed so as to not break the tool for existing users.

[ChristopherMayes]

This file was produced by Bmad. We should discuss this with @davidsagan at https://github.com/bmad-sim/bmad-ecosystem/issues

[DavidSagan]

The BeamPhysics extension for OpenPMD only becomes official with the 2.0 version so making things compatible with V1.1 I'm guessing is not possible.

[ax3l]

That's right, but we found small little defects in what we propose for 2.0 that we would like to fix now, before we standardize them.

The validator for 2.0 already checks all proposed changes and does not pass the example files. https://github.com/openPMD/openPMD-validator/tree/2.0.X

@s-sajid-ali volunteered to update the repo here as part of CAMPA, and we will need only tiny updates (e.g., a few more attributes and a particle species directory) in Bmad to improve it.

[s-sajid-ali]

@ax3l : Could you expand on how the 2.0 standard is a superset of the 1.1 standard which would make every 2.0 compliant OpenPMD file also a 1.1 compliant file (but not the other way around) ?

[DavidSagan]

@s-sajid-ali No. 2.0 compliant will not necessarily be 1.1 compliant.

[ax3l]

I understand the confusion now, @s-sajid-ali, I must have explained it badly, sorry for this.

openPMD extensions, e.g., the beamphysics extension, must be compatible with the base standard (of the respective version). Otherwise we cannot use a lot of tools for the community, e.g., extract particles, convert series, use large vis tools like ParaView, etc.

Currently, the BeamPhysics extension (for 2.0) is not compatible with the openPMD standard (2.0), because we are a bit ambiguous in the wording and forgot the particle species group in /data/%T/particles/ + particleSpecies/ + records, which is an easy fix here, in the standard and Bmad.

The other small detail are a few missing attributes in the example file (see: validator for 2.0).

The other thing I mentioned in person: we will likely update openPMD-api to be able to read 1.1 and 2.0 files (and only be able to write 2.0 files) in coming releases.

[ax3l]

The plan of action I propose is:

1. submit a PR here that slightly updates the reader to take the particle species group into account,
2. as the PR is open, @DavidSagan and us update Bmad to add the particle species group and missing attributes, until openPMD-validator for 2.0 passes,
3. update Bmad and the repo here at the same time,

After that, we do another update to the repo here, using openPMD-api (once it got its 2.0 update itself). (This will then support more file formats here, e.g., ADIOS2.)

[ax3l]

We'll take this on again. @moellep is also interested in this and volunteers to post a PR to update this repo (point 1).

I'll work on the validator updates for 2.0.