search

pdidev / pdi

The PDI Data Interface
https://pdi.dev
BSD 3-Clause "New" or "Revised" License
6 stars 0 forks source link

[decl'HDF5] HDF5 compression #449

Closed jbigot closed 2 months ago

jbigot commented 4 months ago

In GitLab by @mtrochon on Jun 17, 2024, 10:34

I am trying to use compression in Gysela. I modified the yaml file as the image shows, in the output log it's written that Decl'HDF5 does use deflate. But I don't see any changes regarding the time and the output file sizes. image

jbigot commented 4 months ago

Can you provide us with your full yaml & log in text format, please?

jbigot commented 4 months ago

also, can you actually check the compression level of each datacheck as described in https://stackoverflow.com/questions/70202779/determine-if-a-hdf5-file-created-with-h5py-is-compressed-or-not

jbigot commented 4 months ago

In GitLab by @mtrochon on Jul 2, 2024, 09:28

Good morning,

I checked the compression and indeed it works. The problem was that i compile gysela+GNU+PDI which gave a very bad compression (from 1.1T to 1.05T) So, I compare it with gysela+Intel+HDF5, where i got good results (from 1.1T to 500GB) Then, i checked gysela+GNU+HDF5, and I got the same result as the first tests.

So the problem is between GNU and HDF5_compression, and I will look more into it. Sorry for the trouble !

jbigot commented 4 months ago

No worry, thanks for the feedback.