[NFR] Add ADIOS to accepted file formats

[n01r]

Hey,

so I have written a lot of openPMD-favored ADIOS output during my last campaign. It would be cool if the conversion would also work for these files.

Perspectively, it might even be beneficial to go from something like hdf_to_gdf to openpmd_to_gdf since the standard makes sure that you always know how to treat the different backends.

[n01r]

Dammit, I built a serial ADIOS with the option --with-hdf5=$HDF5_ROOT to then get access to the tool bp2h5. Unfortunately, I was not able to just take this short route.

Got an error for bp2h5. So I'll have to report that to the ADIOS team and then convert the file to HDF5 by myself.

[n01r]

This comes to mind as well, where one could possibly benefit from the other:
https://github.com/openPMD/openPMD-api/issues/459

[KseniaBastrakova]

I already have an implementation of reading and writing using openPMD in my local branch, so going to merge branch to master and finished on this week

[n01r]

Sweet! Sounds great! :sparkles: :tada:

[ax3l]

Excellent, using openPMD-api instead of h5py for the file handing will solve this immediately :)

[sbastrakov]

@ax3l yes, that is what the branch @KseniaBastrakova mentioned does.

[KseniaBastrakova]

Now, it works with openPMD files. @n01r , the maximum value of possible dataset is 268435455, because of GDF limitations

[sbastrakov]

@KseniaBastrakova just to clarify, do you mean the max amount of macroparticles to be possibly written to a gdf file? Is it per species or overall?

[KseniaBastrakova]

it's the max amount of macroparticles per species.

[n01r]

the maximum value of possible dataset is 268435455, because of GDF limitations

Ha, that just fits ^^ I have the same order of magnitude of protons that I want to convert - but less than this number. :) Let's see if this works.

[n01r]

Hmm ... doesn't work so easily, seems my files are too old and their attributes are different.

Converting .gdf to .hdf file
Destination .gdf directory not specified. Defaulting to simData_protons_rear_all_30000.gdf
Series constructor called with explicit iteration suggests loading a single file with groupBased iteration encoding. Loaded file is fileBased.
Traceback (most recent call last):
  File "/home/garten70/src/openPMD-converter-GDF/hdf_to_gdf.py", line 471, in <module>
    hdf_to_gdf(args.hdf, args.gdf, args.max_cell, args.species)
  File "/home/garten70/src/openPMD-converter-GDF/hdf_to_gdf.py", line 31, in hdf_to_gdf
    series_hdf = openpmd_api.Series(hdf_file_directory, openpmd_api.Access_Type.read_only)
RuntimeError: Unexpected Attribute datatype for 'timeOffset'

I guess I have too look into this a bit more deeply.

[KseniaBastrakova]

Hmm ... doesn't work so easily, seems my files are too old and their attributes are different.

Converting .gdf to .hdf file
Destination .gdf directory not specified. Defaulting to simData_protons_rear_all_30000.gdf
Series constructor called with explicit iteration suggests loading a single file with groupBased iteration encoding. Loaded file is fileBased.
Traceback (most recent call last):
  File "/home/garten70/src/openPMD-converter-GDF/hdf_to_gdf.py", line 471, in <module>
    hdf_to_gdf(args.hdf, args.gdf, args.max_cell, args.species)
  File "/home/garten70/src/openPMD-converter-GDF/hdf_to_gdf.py", line 31, in hdf_to_gdf
    series_hdf = openpmd_api.Series(hdf_file_directory, openpmd_api.Access_Type.read_only)
RuntimeError: Unexpected Attribute datatype for 'timeOffset'

I guess I have too look into this a bit more deeply.

I reproduced your problem. I think it's a problem with PicOnGPU checkpoints in ADOIS. Also, there is new issue about it: https://github.com/ComputationalRadiationPhysics/picongpu/issues/3119

[n01r]

I am not really trying to convert a checkpoint but a normal data dump from a single particle species but that is probably still the same issue.

[sbastrakov]

@n01r I believe the same writers are used for checkpointing and data dump.

In case the issue is indeed what @ax3l suggested, perhaps it would be possible to either fix it in the files or make a branch of openPMD-API that takes it into account. Not 100% sure tho, needs investigating.

[n01r]

It will probably be easier to fix it in my old files. I will only try to convert a very limited number of them to GPT, so fixing it in the files should be the easiest.

[ax3l]

I would recommend to fix your existing files with the ADIOS1 Numpy API: https://github.com/conda-forge/adios-python-feedstock Can you share such a script here or in https://github.com/ComputationalRadiationPhysics/picongpu/issues/3119 for others as well?

[n01r]

Will do once I get to it - it's more of a side project :)

[ax3l]

You know that this is a ten liner - maybe 20 lines if you feel fancy, just do it ;)