search

c3-time-domain / SeeChange

A time-domain data reduction pipeline (e.g., for handling images->lightcurves) for surveys like DECam and LS4
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

add HDF5 compression for background objects #328

Closed guynir42 closed 4 months ago

guynir42 commented 4 months ago

This should reduce the file sizes of the background images by about x2.

guynir42 commented 4 months ago

Writing compressed data with chunck size of 128x128 with minimal gzip compression factor yields a write time (for two full-sized arrays, one for counts and one for variance) that takes up 45MB instead of 70MB with a write time of 2s instead of 0.5s.