search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
134 stars 51 forks source link

added async write flag for ADIOS2 #1460

Closed guj closed 1 year ago

guj commented 1 year ago

please add a PR description, including links to ADIOS2 docs about the added feature

This PR adds a new env variable OPENPMD_ADIOS2_ASYNC_WRITE. Default = 0. When set to 1, it sets the ADIOS parameter "AsyncWrite" to "on". In which case the BP5 engine will handle write calls asynchronously.

See https://adios2.readthedocs.io/en/latest/engines/engines.html#bp5 for details of BP5 feature "AsyncWrite".

guj commented 1 year ago

Hi @franzpoeschel @ax3l

Any idea why the test fail with hdf5 on Conda_ompi_all fails on ?

guj commented 1 year ago

Can you please document the additional parameter in adios2.rst?

Will do Also, note that in BP5, openPMD will flush to disk by default which makes no sense in combination with async I/O. So, if you add an environment variable for this, it probably makes sense to flip that default in this case.

Yes, I had to flip the PerformDataWrite. I wonder whether this should an option as well.

franzpoeschel commented 1 year ago

Yes, I had to flip the PerformDataWrite. I wonder whether this should an option as well.

I think that we don't need an extra option for this. I added a commit that sets the default flush target to the Buffer when using this env var.

guj commented 1 year ago

LGTM The failing docs PR run concerns all PRs currently and has nothing to do with this PR.

Yes, I had to flip the PerformDataWrite. I wonder whether this should an option as well.

I think that we don't need an extra option for this. I added a commit that sets the default flush target to the Buffer when using this env var.

That will do. Thanks.