search

ComputationalRadiationPhysics / jungfrau-photoncounter

Conversion of Jungfrau pixel detector data to photon count rate
GNU General Public License v3.0
2 stars 2 forks source link

Use compression #22

Closed kloppstock closed 6 years ago

kloppstock commented 7 years ago

Use compression on the photon maps.

lopez-c commented 6 years ago

Reducing the amount of data that needs to be stored for further analysis is a main point, even at the cost of performance in terms of maximum throughput. A good starting point to decide which lossless compression algorithm we can use could be based on HZDR research paper.

Would it be possible to get a copy of it?

kloppstock commented 6 years ago

The HZDR did some research concerning the compression factor and the cost of different algorithms with different hardware. The paper can be found here or here.

Recommended file formats / libraries are HDF5 and ADIOS (and ADIOS2). Both should support a wide variety of compression algorithms.

kloppstock commented 6 years ago

As discussed in the meeting on 25. May, this will not be implemented in the photon counting project itself.