search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

Loss of calibration provenance in processed science data #2012

Open akremin opened 1 year ago

akremin commented 1 year ago

When we process a science exposure, we use four types of calibration files that are typically derived from calibration data taken on the night of observations: badcolumns, biasnight, psfnight, and fiberflatnight. The nightly calibrations are stored in $DESI_SPECTRO_REDUX/$SPECPROD/calibnight/<NIGHT>. If the files for a given camera on a given night are not present, then we use default versions of these data in an SVN directory.

Since the choice of calibration file is dynamically selected at runtime based on whether the data exists from a given night or not, we should be saving which file was used in the header of the processed files.

We do trace the origins of the psf that we use: In the 'FLUX' HDU of frame, cframe, etc. it is saved as the header keyword ORIG_PSF. In the 'PSF' HDU of the psf files it is saved under the header keyword IN_PSF.

The other file types are not propagated to any of the science exposure headers. We should add these.

sbailey commented 1 year ago

For the record: the information is in the log files for the jobs so it is still possible to figure out what flat was used, but I completely agree that this provenance information should also be in the headers of the data files themselves.