Closed SimonKrughoff closed 6 years ago
I've tracked down to references on FITS compression: https://community.lsst.org/t/fits-tile-compression-support/2233 https://community.lsst.org/t/lossless-fits-compression-enabled/2410
The second states that lossless compression is activated by default. The first suggests that compression could be turned off with something like:
binned_sensor_fits_halves:
recipe: noCompression
...
This is based on writeRecipes.yaml.
The storage model we have for the Stack outputs at NERSC assumes, at a minimum, the default compression currently implemented. Is there no way to handle the compressed data on the image-viewer side?
Ok, I've made this change to our policy config file and confirmed that it removes compression. Change is committed and pushed.
@jchiang87 is storage an issue? These images are downsampled by a factor 16 and we were thinking about only generating them for < 10% of DC2.
The image viewer handles RICE compressed images, so we could potentially add a new recipe to writeRecipes.yaml
. (Note that the lossyBasic
recipe doesn't work because it gzips the mask plane.) For DES we used globally fpacked images, but I think tile compression should be ok too.
It wasn't clear where you were intending to turn off the compression. From your initial comment it sounded like all compression for all data products coming from the Stack was going to be turned off. If it's just the derived down-sampled images, then indeed I don't care.
My intention was just the downsampled images. @SimonKrughoff should confirm.
@kadrlica is this PR ready to merge? The DM-DC2 folks are interested in having one consistent version of obs_lsstSim, and it seems we are interested in these particular updates for use in the "mini-DRP".
@heather999 I think @SimonKrughoff and I decided that these changes didn't require another review, but I'd be happy to perform one if it was deemed necessary.
Instructions for running at NERSC: