search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
134 stars 51 forks source link

Initial support for HDF5 subfiling #1580

Closed franzpoeschel closed 5 months ago

franzpoeschel commented 6 months ago

I'm following the documentation in this presentation.

TODO:

ax3l commented 5 months ago

cc @guj and @jeanbez : just FYI, sub-filing support for HDF5 in openPMD-api now :)

ax3l commented 5 months ago

We should check if Conda-Forge (hdf5 and h5py), h5py (pip builds) and Spack enable this feature in the HDF5 package already (non-default).

franzpoeschel commented 5 months ago

Note that this needs to be enabled explicitly when configuring HDF5 -DHDF5_ENABLE_SUBFILING_VFD=ON.

guj commented 5 months ago

Is MPI_THREAD_MULTIPLE. still a requirement to use subfiles for HDF5? If so, needs to state in the doc.

franzpoeschel commented 5 months ago

Is MPI_THREAD_MULTIPLE. still a requirement to use subfiles for HDF5? If so, needs to state in the doc.

Yes, it is a requirement. It's mentioned in the docs (Note that the subfiling VFD needs to be enabled explicitly when configuring HDF5 and threaded MPI must be used.), though the specific level is not mentioned yet (MPI_THREAD_MULTIPLE).