search

F2I-Consulting / fesapi

API for ENERGISTICS™ data standards (mainly RESQML™), multi-languages (C++, Java, C#, Python)
Apache License 2.0
34 stars 24 forks source link

Use hdf5 compression level with MPI #332

Closed untereiner closed 9 months ago

untereiner commented 11 months ago

Is your feature request related to a problem? Please describe. I need to lower the memory consumption of the hdf5 files.

Describe the solution you'd like I want to compress the hdf data written by each rank. Not sure if it requires now code. But I can't figure out how to manage with existing code. Because it should involve chunking, collective MPI operations. But I am not sure it need hyperslabing.

Describe alternatives you've considered n.a.

Additional context n.a.

philippeVerney commented 11 months ago

Hi Lionel,

This is defintiely a good idea but yes it requires new code and thought/study ; it is not straightforward. Basically it is the general idea on how to port HDF5 functionalities (but not all of them because we focus on Energistics) + how does it behave in an ETP context. I think hyperslabbing is needed as well because you want to read/write according to your defined chunks (but maybe nothing new compared as what is available today in FESAPI)

Se, definitely a valid issue. But I cannot tell you when I'll have time to work on it. In any case thanks because it is a valuable ticket in the backlog.

philippeVerney commented 11 months ago

It would also break FETPAPI compatibility since we are going to change the HDF Proxy prototypes. Not a big deal but a bit ennoying.

untereiner commented 11 months ago

I need it so I will try to work on it