ADIOS Support

[ax3l]

We need an additional module to read ADIOS files.

Parallel write and read python bindings for ADIOS are available (use ADIOS 1.8.0+).

The API for the picongpu module should be able to transparently switch between the two formats, whatever is provided (our meta information and hierarchy is the same in both file formats)

Compile notes

• download and install mxml first
• install adios example with configure options (remove the initial sudo's and add the python binding flags to configure)
• also depends on MPI (such as parallel hdf5 does)

[ax3l]

I added an ADIOS file on hypnos at /data/hplsim/scratch/huebl/lwfa_adios003_3D.

Unfurtunately, our ADIOS plugin still lacks 2D support but I enabled zlib compression :)

during adios ./configure, don't forget to set --with-zlib=/usr on hypnos.

update:

• [x] ah, files might be damaged. pls check /data/hplsim/scratch/huebl/bunch_adios002 instead

[ax3l]

@Heikman great you finished the implementation!

The adios py-wrapper got some update from Jong Youl Choi in the last days: it is now possible to to read directly with

# old
v = f.var['temperature']
v.read(offset=(0,5), count=(2, 5))

# now
v = f['temperature']
v[0:2,5:10]

-> striding is not (yet) supported since that can cause heavy seek-ing and needs further investigation to be run in a performant way.

Further features:

• reading attributes in now implemented (via the independent list in f.attr['......'])

Coming but still missing features:

• reading attributes and values in an "object-oriented way" (not yet decided, we need a work-around for now or at least pls forward the attributes transparently to the pyDive user)
• adios python write API (in the making)