desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

Review how mask HDUs are compressed #546

Closed weaverba137 closed 6 years ago

weaverba137 commented 6 years ago

In reviewing the data model for release 18.3, I've found several files that have FITS-compressed HDUs. These HDUs are all masks:

The compression does not appear to be applied consistently on these files. In particular, the spectra-NSIDE files may be adding extraneous header keywords that may cause astropy.io.fits to interpret the compressed HDUs as compressed primary HDUs, instead of compressed extension HDUs (see astropy/astropy#7312).

In addition, fiberflat and frame files use RICE_1, while spectra-NSIDE files use GZIP_1. Is there a reason for this choice? All of the masks are 2D images, so the dimensionality is not the reason.

Note, I don't think this issue needs to be resolved for release 18.3, but it is something that could bite us in the future.

weaverba137 commented 6 years ago

@julienguy, I think I saw another issue related to this, and I think the conclusion was that these mask HDUs should not be compressed at all. Could you please comment?

weaverba137 commented 6 years ago

@julienguy, never mind, it's #603 I was thinking of.