HDFGroup / hermes

Extending the HDF5 library to support intelligent I/O buffering for deep memory and storage hierarchy systems
https://grc.iit.edu/research/projects/hermes
30 stars 17 forks source link

Investigate Hermes VOL #636

Open lukemartinlogan opened 8 months ago

lukemartinlogan commented 8 months ago

There has been interest in transparent dynamic data tagging in Hermes. However, this requires advanced knowledge of data format, which is difficult to know in a VFD. We will likely need a VOL.

We have a scientist with climate data. Some data has temperature. Humidity and pressure of ocean. Scientist in the future is always interested in searching regions. Regions with extremely high temperature

Scientist created the dataset Give the data management system something to track

Let's say we have an app with HDF5. HDF5 writes a 3D array. HDF5 calls Hermes. How Hermes divides the 3D array? The user interacts with HDF5. How does the user pass the intention through the HDF5 API.