search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
142 stars 51 forks source link

Obtain chunk offset, extent without loading it #993

Open eschnett opened 3 years ago

eschnett commented 3 years ago

I am looking for a function to determine a chunk's offset and extent without loading the chunk. This would allow me to create the grid structure in AMReX, and then schedule the chunk loads into AMReX's storage.

ax3l commented 3 years ago

The function you need is RecordComponent::availableChunks: https://github.com/openPMD/openPMD-api/blob/3ee8b57dd8fa2d1559f5ee85ba08f5026db60be0/include/openPMD/backend/BaseRecordComponent.hpp#L62-L80

Examples in our tests: https://github.com/openPMD/openPMD-api/blob/3ee8b57dd8fa2d1559f5ee85ba08f5026db60be0/test/SerialIOTest.cpp#L183-L202 or https://github.com/openPMD/openPMD-api/blob/3ee8b57dd8fa2d1559f5ee85ba08f5026db60be0/test/ParallelIOTest.cpp#L336-L339

It's newly implemented in ADIOS2 and we can in theory also implement it for HDF5, although with a somewhat different meaning there.

I just realized we only show this in tests so far, but not in user-facing examples. I also make use of chunk meta-data reading in the DASK bindings in the development branch: https://github.com/openPMD/openPMD-api/tree/dc71272a3848d841829d9f0ff14af61dece9c18e/src/binding/python/openpmd_api

eschnett commented 3 years ago

Should RecordComponent::availableChunks be const?

ax3l commented 3 years ago

From the user-facing API yes, but it's possible that we cannot mark it const without mutable tricks because we do meta data reads under the hood (we interact with IOHandler that is often a member).