search

openPMD / openPMD-api

:floppy_disk: C++ & Python API for Scientific I/O
https://openpmd-api.readthedocs.io
GNU Lesser General Public License v3.0
142 stars 51 forks source link

Parameter Read/Write: Data Policy #470

Open ax3l opened 5 years ago

ax3l commented 5 years ago

It would probably be useful if we added a trait / policy or runtime converter to the Parameter< Operation::READ_DATASET > (and write) data members of shared pointers in order to share more complex data objects.

For example returning to python a py::array, that the user might not handle, might result in it getting garbage collected before our flush() can be called, which in turn causes in invalid memory access in the backend. We could keep that py::array alive until flush() as in C++ by returning a shared_ptr< py::array >.

Overloads for RecordComponent::loadChunk<T> are already okay and well designed, but we cannot share yet an object that needs a more indirect data access pattern than a shared_ptr of raw memory, since we access it right away: https://github.com/openPMD/openPMD-api/blob/0.7.1-alpha/src/IO/HDF5/HDF5IOHandler.cpp#L1017

ax3l commented 5 years ago

cc @C0nsultant @franzpoeschel do you have thoughts on this? We could improve data safety for user-provided containers and python bindings a lot if we could store more complex T types in our std::shared_ptr<T> data members.

ax3l commented 4 years ago

Some first support for 1D contiguous containers: #204 ADIOS2 has an interface providing C++20 std::span and we could also think about mdspan.