search

ascot4fusion / ascot5

ASCOT5 is a high-performance orbit-following code for fusion plasma physics and engineering
https://ascot4fusion.github.io/ascot5/
GNU Lesser General Public License v3.0
31 stars 9 forks source link

Add compression to marker input hdf5 writing routines #14

Closed sjjamsa closed 1 year ago

sjjamsa commented 1 year ago

We probably should merge this to develop after feature/8 has been merged there.

miekkasarki commented 1 year ago

I've no issues with this PR just a few questions:

  1. Can we expect a significant performance hit-back in post-processing? If so, then this should be guarded by a compiler flag.
  2. I though datasets in HDF5 were compressed already?
  3. If the datasets are not compressed, should we consider compressing output dists and input magnetic field / wall as well? (Not in this PR of course)

As a side note: future development would be to sample markers from a distribution run-time and store wall loads directly (W/m^2) so that storing ini/endstate would be optional and instead of marker input we would have an input distribution(s).

sjjamsa commented 1 year ago

You need to specifically request compression filter to compress a dataset. It requires chunking of the dataset. It is all transparent after you request it. I suppose the performance hit could be there if we have many small datasets and each need to go through the decompression. I bet the small datasets could simply increase in size instead of shrink. In case of large datasets, I would rather expect improved performance, as there is less to read from slow disk.

sjjamsa commented 1 year ago

This pull request was sort of a miss. Removing it.