Issue reading converted files using openPMD

[jhornung-gh]

Hi,

I have issues reading the converted files with openPMD version 0.15.2. The "non-converted" openPMD files from PIConGPU work fine, however I did not test it using HDF5 files from PIConGPU. I tested it within the PP-Container.

When trying to open the files via a series using the Adios Backend I get the following Error:

[ADIOS2] Warning: Attribute with name /data/0/dt has no type in backend. Cannot read iteration '0' and will skip it due to read error: Read Error in backend ADIOS2 Object type: Attribute Error type: NotFound Further description: /data/0/dt

Another error occurs when reading the converted HDF5 output:

_HDF5-DIAG: Error detected in HDF5 (1.14.2) thread 0:

000: H5G.c line 442 in H5Gopen2(): unable to synchronously open group

major: Symbol table
minor: Unable to create file

001: H5G.c line 403 in H5G__open_api_common(): unable to open group

major: Symbol table
minor: Can't open object

002: H5VLcallback.c line 4545 in H5VL_group_open(): group open failed

major: Virtual Object Layer
minor: Can't open object

003: H5VLcallback.c line 4512 in H5VL__group_open(): group open failed

major: Virtual Object Layer
minor: Can't open object

004: H5VLnative_group.c line 154 in H5VL__native_group_open(): unable to open group

major: Symbol table
minor: Can't open object

005: H5Gint.c line 379 in H5G__open_name(): group not found

major: Symbol table
minor: Object not found

006: H5Gloc.c line 421 in H5G_loc_find(): can't find object

major: Symbol table
minor: Object not found

007: H5Gtraverse.c line 816 in H5G_traverse(): internal path traversal failed

major: Symbol table
minor: Object not found

008: H5Gtraverse.c line 596 in H5G__traverse_real(): traversal operator faile d

major: Symbol table
minor: Callback failed

009: H5Gloc.c line 381 in H5G__loc_find_cb(): object '0' doesn't exist

major: Symbol table
minor: Object not found

[AbstractIOHandlerImpl] IO Task OPENPATH failed with exception. Clearing IO que ue and passing on the exception. Cannot read iteration '0' and will skip it due to read error: Read Error in backend HDF5 Object type: Group Error type: NotFound Further description: [HDF5] Internal error: Failed to open HDF5 group during path opening

I looks like this is somewhat connected to the conversion but I am not 100% sure. You can try to reproduce it using my readout file in "/lustre/phx/jhornung/openpmd/read_openpmd_2d.py" and the data in "/lustre/phx/jhornung/sdf2opmd/sim"

[denisbertini]

interesting, i can without problem read the converted hdf5 with openpmd-api in c++:

[dbertini@lxbk1131 ~/gsi_git/sdf2opmd-build]$ opmd_reader_2d -f /lustre/rz/dbertini/sdf2opmd/sim/epoch2d/data/0014.h5
filename: /lustre/rz/dbertini/sdf2opmd/sim/epoch2d/data/0014.h5
Read iterations     1
 Read attributes in the root:
    author
    basePath
    date
    iterationEncoding
    iterationFormat
    meshesPath
    openPMD
    openPMDextension
    particlesPath
    software
    softwareVersion

basePath - /data/%T/
iterationEncoding - groupBased
iterationFormat - /data/%T/
meshesPath - meshes/
openPMD - 1.1.0
openPMDextension - 0
particlesPath - particles/

Read attributes in basePath:

Read iterations in basePath:
    1

Read attributes in iteration 1:
    dt
    time
    timeUnitSI

1.time - 0
1.dt - 1
1.timeUnitSI - 1

Read attributes in meshesPath in iteration 1:

Read meshes in iteration 1:
    charge_density
    ex
    ey
    ez
    grid_x
    grid_y
    number_density
    number_density_electron_l
    number_density_electron_r

[denisbertini]

humm using c++ interface, is of course not python but it is using the same code to read ...

[denisbertini]

this is the reader c++ that i used :

int main(int argc, char *argv[])
{

    int opt;
    char* fname=NULL;

    while((opt = getopt(argc, argv, ":f:x")) != -1) 
    { 
        switch(opt) 
        { 
            case 'x': 
            case 'f': 
                printf("filename: %s\n", optarg);
        fname=optarg;
                break; 
        } 
    }   

    // Initiate the reading procedure

    getfullInfo(fname);

    std::cout << " Now create a serie " << std::endl;

    // Create a series and open the file
    Series series = Series( fname, Access::READ_ONLY);

    cout << "Read a Series with openPMD standard version "
         << series.openPMD() << '\n';
    cout << "The Series contains " << series.iterations.size() << " iterations:"; 

    // Loop over all iterations in the file
    int iter=0;
    for( auto const& i : series.iterations ){
      // Meshes  
      cout << "Iteration " << iter << " contains " << i.second.meshes.size() << " meshes:";
      for( auto const& m : i.second.meshes )
        cout << "\n\t" << m.first;

      cout << '\n';
      cout << "Iteration: "  <<  iter << "contains " << i.second.particles.size() << " particle species:";

      // Loop over species
      for( auto const& ps : i.second.particles ) {
        cout << "\n\t" << ps.first;
      }
      cout << '\n'; 

      iter++;
      Iteration j = i.second;

      // Particles Species      
      openPMD::ParticleSpecies deuterons = j.particles["electron_l"];
      //std::shared_ptr<double> charge = electrons["charge"][openPMD::RecordComponent::SCALAR].loadChunk<double>();
      series.flush();

      // Access attribute within sub-group electron_gridx
      for( auto const& a : deuterons["momentum"].attributes() ){
    std::cout << '\t' << a << '\n'; 
      }
      std::cout << '\n';

      auto p_x = deuterons["momentum"]["x"];
      Extent g_extent = p_x.getExtent();
      auto all_x  = p_x.loadChunk<double>();
      series.flush();

      cout << "Full Electron gridx starts with:\n\t{";
      for( size_t col = 0;  col < 5; ++col )
    cout << all_x.get()[col] << ", ";

      cout << "...}\n";

    }//!iteration++ 

    return 0;
}

[denisbertini]

In this example, all is used basically

• Series creation
• series.flush() calls
• loadChunk() to get full mesh data ( here in 2D )

[denisbertini]

output that i get is OK:

dbertini@lxbk1131 ~/gsi_git/sdf2opmd-build]$ opmd_reader_2d -f /lustre/rz/dbertini/sdf2opmd/sim/epoch2d/data/0014.h5
filename: /lustre/rz/dbertini/sdf2opmd/sim/epoch2d/data/0014.h5
Read iterations     1
 Read attributes in the root:
    author
    basePath
    date
    iterationEncoding
    iterationFormat
    meshesPath
    openPMD
    openPMDextension
    particlesPath
    software
    softwareVersion

basePath - /data/%T/
iterationEncoding - groupBased
iterationFormat - /data/%T/
meshesPath - meshes/
openPMD - 1.1.0
openPMDextension - 0
particlesPath - particles/

Read attributes in basePath:

Read iterations in basePath:
    1

Read attributes in iteration 1:
    dt
    time
    timeUnitSI

1.time - 0
1.dt - 1
1.timeUnitSI - 1

Read attributes in meshesPath in iteration 1:

Read meshes in iteration 1:
    charge_density
    ex
    ey
    ez
    grid_x
    grid_y
    number_density
    number_density_electron_l
    number_density_electron_r

 Now create a serie 
Read a Series with openPMD standard version 1.1.0
The Series contains 1 iterations:Iteration 0 contains 9 meshes:
    charge_density
    ex
    ey
    ez
    grid_x
    grid_y
    number_density
    number_density_electron_l
    number_density_electron_r
Iteration: 0contains 2 particle species:
    electron_l
    electron_r
    timeOffset
    unitDimension

Full Electron gridx starts with:
    {-2.32036e-24, -2.4846e-24, -2.46624e-24, -2.40357e-24, -2.4052e-24, ...}

[jhornung-gh]

Did you also try with my converted data?

[denisbertini]

no that is what i wanted to do, where can i find your converted files?

[jhornung-gh]

/lustre/phx/jhornung/sdf2opmd/sim

[denisbertini]

wow ... it crashes even reading just the files attributes ...

[dbertini@lxbk1131 ~/gsi_git/sdf2opmd-build]$ opmd_reader_2d -f /lustre/phx/jhornung/sdf2opmd/sim/simData_0000.h5
filename: /lustre/phx/jhornung/sdf2opmd/sim/simData_0000.h5
Read iterations     1
 Read attributes in the root:
    author
    basePath
    date
    iterationEncoding
    iterationFormat
    meshesPath
    openPMD
    openPMDextension
    software
    softwareVersion

basePath - /data/%T/
iterationEncoding - groupBased
iterationFormat - /data/%T/
meshesPath - meshes/
openPMD - 1.1.0
openPMDextension - 0
terminate called after throwing an instance of 'openPMD::error::NoSuchAttribute'
  what():  particlesPath
Aborted (core dumped)

[jhornung-gh]

Maybe because I did not convert everything? I think I only got ey and density data

[denisbertini]

Look: In my case the problematic particlesPath in included in the root of the HDF file as a section-item:

Read attributes in the root:
    author
    basePath
    date
    iterationEncoding
    iterationFormat
    meshesPath
    openPMD
    openPMDextension
    particlesPath
    software
    softwareVersion

but not in your case, it is missing and the reader crashes trying to get it, NoSuchAttribute

[denisbertini]

ah indeed the particles are missing here ... then my reading procedure is wrong

[denisbertini]

i should comment out all the particle reading part

[denisbertini]

commenting this out, it works through! until it crashes but this is clear, it try to retrieve particle info that does not exist in your file

[denisbertini]

[dbertini@lxbk1131 ~/gsi_git/sdf2opmd-build]$ opmd_reader_2d -f /lustre/phx/jhornung/sdf2opmd/sim/simData_0000.h5
filename: /lustre/phx/jhornung/sdf2opmd/sim/simData_0000.h5
Read iterations     1
 Read attributes in the root:
    author
    basePath
    date
    iterationEncoding
    iterationFormat
    meshesPath
    openPMD
    openPMDextension
    software
    softwareVersion

basePath - /data/%T/
iterationEncoding - groupBased
iterationFormat - /data/%T/
meshesPath - meshes/
openPMD - 1.1.0
openPMDextension - 0

Read attributes in basePath:

Read iterations in basePath:
    1

Read attributes in iteration 1:
    dt
    time
    timeUnitSI

1.time - 0
1.dt - 1
1.timeUnitSI - 1

Read attributes in meshesPath in iteration 1:

Read meshes in iteration 1:
    charge_density
    ex
    ey
    grid_x
    grid_y
    number_density

 Now create a serie 
Read a Series with openPMD standard version 1.1.0
The Series contains 1 iterations:Iteration 0 contains 6 meshes:
    charge_density
    ex
    ey
    grid_x
    grid_y
    number_density
Iteration: 0contains 0 particle species:
terminate called after throwing an instance of 'std::out_of_range'
  what():  Key 'electron_l' does not exist (read-only).
Aborted (core dumped)

[denisbertini]

so i can read basically your file

[jhornung-gh]

Can you also try it with the python api? You can just use my script, located in /lustre/phx/jhornung/openpmd/read_openpmd_2d.py

[denisbertini]

you are reading my mind !

[denisbertini]

that what i just wanted to do now

[denisbertini]

i modified your python script and all works fine

dbertini@lxbk1131 /lustre/rz/dbertini/sdf2opmd/analysis]$ ls -al
total 10
drwxr-sr-x. 2 dbertini rz  4096 Sep 12 15:09 .
drwxr-sr-x. 7 dbertini rz  4096 Sep 12 15:09 ..
-rw-r--r--. 1 dbertini rz 56445 Sep 12 15:09 e_density-1.png
-rw-r--r--. 1 dbertini rz 54965 Sep 12 15:09 field_Ex-1.png

[denisbertini]

i could also reproduce the terrible errors of this issue because the path+filename was wrong and pointing to nowehere:

[dbertini@lxbk1131 /lustre/rz/dbertini/sdf2opmd/ana/johannes]$ python3 read_openpmd_2d.py
/lustre/rz/dbertini/sdf2opmd/ana/johannessimData_0001.h5
HDF5-DIAG: Error detected in HDF5 (1.14.2) thread 0:
  #000: H5F.c line 836 in H5Fopen(): unable to synchronously open file
    major: File accessibility
    minor: Unable to open file
  #001: H5F.c line 796 in H5F__open_api_common(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #002: H5VLcallback.c line 3863 in H5VL_file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #003: H5VLcallback.c line 3675 in H5VL__file_open(): open failed
    major: Virtual Object Layer
    minor: Can't open object
  #004: H5VLnative_file.c line 128 in H5VL__native_file_open(): unable to open file
    major: File accessibility
    minor: Unable to open file
  #005: H5Fint.c line 1853 in H5F_open(): unable to open file: name = '/lustre/rz/dbertini/sdf2opmd/ana/johannessimData_0001.h5', tent_flags = 0
    major: File accessibility
    minor: Unable to open file
  #006: H5FD.c line 769 in H5FD_open(): open failed
    major: Virtual File Layer
    minor: Unable to initialize object
  #007: H5FDsec2.c line 326 in H5FD__sec2_open(): unable to open file: name = '/lustre/rz/dbertini/sdf2opmd/ana/johannessimData_0001.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0
    major: File accessibility

that was the problem of your script at this line

# file name or iterator (%T) name. Changing this to %T will iterate over all available .bp files.
file_name = 'simData_%T.h5'

This is an openPMD flag %T which will not be interpreted in python

[jhornung-gh]

How come I am able to read any data when the %T is the issue? I would have expected that it does not read anything at all. What PP container are you using? I am using: /lustre/rz/dbertini/containers/prod/rlx8_ompi_ucx.sif

[denisbertini]

i am using the same container right now

[denisbertini]

I don't know how you could read any data with that symbol in you filename definition, pure mystery, For me it just can not work ....

[denisbertini]

the modified python scripts can be found in

[dbertini@lxbk1131 /lustre/rz/dbertini/sdf2opmd/ana/johannes]$ ls
read_openpmd_2d.py  read_openpmd_2d.py~  simData_0001.h5

[jhornung-gh]

But the %T is excactly how they describe it in the openPMD docs on the python-api.

https://openpmd-api.readthedocs.io/en/0.15.2/usage/firstread.html

Also, I am able to read the PIConGPU Adios output using the %T format.

[denisbertini]

it does not seem to work for the converted files then ... for a reason that i do not know

[denisbertini]

the converter uses exactly the writing syntax used also in the openPMD docs, it writes files using the Series object entity with auto iteration encoding https://openpmd-api.readthedocs.io/en/0.15.2/usage/serial.html the only difference is that in the converter case the MPI_COMM_WORLD is used :+1:

// Creating series
  Series series= Series(hdf_file.c_str(), Access::CREATE, MPI_COMM_WORLD);

May be the reason ?

[denisbertini]

i change back to data%T and this is what's i get:

dbertini@lxbk1131 /lustre/rz/dbertini/sdf2opmd/ana/johannes]$ python3 read_openpmd_2d.py 
analysis path already exists
/lustre/rz/dbertini/sdf2opmd//ana/johannes/simData_%T.h5
Iteration:  1
[AbstractIOHandlerImpl] IO Task READ_DATASET failed with exception. Clearing IO queue and passing on the exception.
Traceback (most recent call last):
  File "read_openpmd_2d.py", line 169, in <module>
    series.flush()  
RuntimeError: bad optional access

[jhornung-gh]

The error is the same afterwards regarding the "flush()" command. I also got this error before. How come you are not getting the other HDF5 error anymore?

[jhornung-gh]

Ah yeah, you are only using one .h5 file. The HDF5 error only occurs once you got more than one file.

[denisbertini]

seems not trivial to understand that, just one question : do you need this functionality ?

[denisbertini]

giving full filename works ok ...

[denisbertini]

It will need some iteration with the openPMD api developper to solve that i am afraid

[jhornung-gh]

It is not a show-stopper if this is not included, so I guess we can keep it as it is for now and the work-around is not that difficult. Thanks for helping!

[denisbertini]

we need to install a openPMD-api and python 3.8 on one linux without container, and test there if the functionality works. I can do that tomorrow, i am at GSI ... It can be a side effect on working from within a container.

[denisbertini]

The issue with %T flag should be now solved with the commit 0ea8b0672bf0b17a4998f27ffe9428688eb82c82