SOX-062-A08: Record input filenames in product FITS headers

thespacedoctor commented 2 years ago

SOX-062-A08

We also noted that products do not contain the list of input RAW (and calib) data that were used to create them. This is critical if one would like to do QC, debug and can be used to define a product frame provenance (reconstruct the set of data that were used in the data reduction chain to generate a given data product).

[ ] read the Phase 3 data standards to remind myself of the format needed to record provenance
[ ] add the writing to the FITS headers of this provenance when the product is saved
[ ] also add the recipe settings to the FITS header
[ ] update feedback response doc

thespacedoctor commented 2 years ago

The XShooter pipeline reduced MBIAS frame has the following in the header:

HIERARCH ESO PRO REC1 ID = 'xsh_mbias' / Pipeline recipe (unique) identifier
HIERARCH ESO PRO REC1 DRS ID = 'cpl-7.1.1' / Data Reduction System identifier
HIERARCH ESO PRO REC1 PIPE ID = 'xshoo/3.3.10' / Pipeline (unique) identifier
HIERARCH ESO PRO REC1 RAW1 NAME = 'XSHOO.2019-08-23T11:02:26.629.fits' / File na
HIERARCH ESO PRO REC1 RAW1 CATG = 'BIAS_UVB' / Category of raw frame
HIERARCH ESO PRO REC1 RAW2 NAME = 'XSHOO.2019-08-23T11:03:58.817.fits' / File na
HIERARCH ESO PRO REC1 RAW2 CATG = 'BIAS_UVB' / Category of raw frame
HIERARCH ESO PRO REC1 RAW3 NAME = 'XSHOO.2019-08-23T11:05:33.525.fits' / File na
HIERARCH ESO PRO REC1 RAW3 CATG = 'BIAS_UVB' / Category of raw frame
HIERARCH ESO PRO REC1 RAW4 NAME = 'XSHOO.2019-08-23T11:07:06.974.fits' / File na
HIERARCH ESO PRO REC1 RAW4 CATG = 'BIAS_UVB' / Category of raw frame
HIERARCH ESO PRO REC1 RAW5 NAME = 'XSHOO.2019-08-23T11:08:40.772.fits' / File na
HIERARCH ESO PRO REC1 RAW5 CATG = 'BIAS_UVB' / Category of raw frame
HIERARCH ESO PRO REC1 CAL1 NAME = 'XS_GBPM_091001B_UVB_1x2.fits' / File name of
HIERARCH ESO PRO REC1 CAL1 CATG = 'BP_MAP_RP_UVB' / Category of calibration fram
HIERARCH ESO PRO REC1 CAL1 DATAMD5 = '74af5f9ae190c91ec4b7251ebc8a5068' / MD5 si
HIERARCH ESO PRO REC1 PARAM1 NAME = 'keep-temp' / If 'no', temporary files are d
HIERARCH ESO PRO REC1 PARAM1 VALUE = 'no      ' / Default: 'no'
HIERARCH ESO PRO REC1 PARAM2 NAME = 'debug-level' / Additional xshooter debug le
HIERARCH ESO PRO REC1 PARAM2 VALUE = 'none    ' / Default: 'none'
HIERARCH ESO PRO REC1 PARAM3 NAME = 'time-stamp' / Add timestamp to product file
HIERARCH ESO PRO REC1 PARAM3 VALUE = 'false   ' / Default: false
HIERARCH ESO PRO REC1 PARAM4 NAME = 'decode-bp' / Integer representation of the
HIERARCH ESO PRO REC1 PARAM4 VALUE = '2144337919' / Default: 2144337919
HIERARCH ESO PRO REC1 PARAM5 NAME = 'stack-method' / Method used to build master
HIERARCH ESO PRO REC1 PARAM5 VALUE = 'median  ' / Default: 'median'
HIERARCH ESO PRO REC1 PARAM6 NAME = 'klow    ' / Kappa used to clip low level va
HIERARCH ESO PRO REC1 PARAM6 VALUE = '5       ' / Default: 5
HIERARCH ESO PRO REC1 PARAM7 NAME = 'khigh   ' / Kappa used to clip high level v
HIERARCH ESO PRO REC1 PARAM7 VALUE = '5       ' / Default: 5
HIERARCH ESO PRO REC1 PARAM8 NAME = 'fpn_llx ' / Lower left reference area X coo
HIERARCH ESO PRO REC1 PARAM8 VALUE = '10      ' / Default: 10
HIERARCH ESO PRO REC1 PARAM9 NAME = 'fpn_lly ' / Lower left reference area Y coo
HIERARCH ESO PRO REC1 PARAM9 VALUE = '10      ' / Default: 10
HIERARCH ESO PRO REC1 PARAM10 NAME = 'fpn_urx ' / Upper right reference area X c
HIERARCH ESO PRO REC1 PARAM10 VALUE = '1024    ' / Default: 1024
HIERARCH ESO PRO REC1 PARAM11 NAME = 'fpn_ury ' / Upper right reference area Y c
HIERARCH ESO PRO REC1 PARAM11 VALUE = '1024    ' / Default: 1024
HIERARCH ESO PRO REC1 PARAM12 NAME = 'fpn_hsize' / Sample size for Fixed Pattern
HIERARCH ESO PRO REC1 PARAM12 VALUE = '10      ' / Default: 10
HIERARCH ESO PRO REC1 PARAM13 NAME = 'fpn_nsamples' / Number of sampling points
HIERARCH ESO PRO REC1 PARAM13 VALUE = '100     ' / Default: 100
HIERARCH ESO PRO REC1 PARAM14 NAME = 'ron_method' / RON computation method
HIERARCH ESO PRO REC1 PARAM14 VALUE = 'ALL     ' / Default: 'ALL'
HIERARCH ESO PRO REC1 PARAM15 NAME = 'random_sizex' / Region X size for random c
HIERARCH ESO PRO REC1 PARAM15 VALUE = '10      ' / Default: 10
HIERARCH ESO PRO REC1 PARAM16 NAME = 'random_nsamples' / Number of random sample
HIERARCH ESO PRO REC1 PARAM16 VALUE = '100     ' / Default: 100
HIERARCH ESO PRO REC1 PARAM17 NAME = 'ref1_llx' / Lower left X of reference regi
HIERARCH ESO PRO REC1 PARAM17 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM18 NAME = 'ref1_lly' / Lower left Y of reference regi
HIERARCH ESO PRO REC1 PARAM18 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM19 NAME = 'ref1_urx' / Upper right X of reference reg
HIERARCH ESO PRO REC1 PARAM19 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM20 NAME = 'ref1_ury' / Upper right Y of reference reg
HIERARCH ESO PRO REC1 PARAM20 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM21 NAME = 'ref2_llx' / Lower left X of reference regi
HIERARCH ESO PRO REC1 PARAM21 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM22 NAME = 'ref2_lly' / Lower left Y of reference regi
HIERARCH ESO PRO REC1 PARAM22 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM23 NAME = 'ref2_urx' / Upper right X of reference reg
HIERARCH ESO PRO REC1 PARAM23 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM24 NAME = 'ref2_ury' / Upper right Y of reference reg
HIERARCH ESO PRO REC1 PARAM24 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM25 NAME = 'stacking_ks_low' / Lower value of kappa-si
HIERARCH ESO PRO REC1 PARAM25 VALUE = '3       ' / Default: 3
HIERARCH ESO PRO REC1 PARAM26 NAME = 'stacking_ks_iter' / Number of iterations i
HIERARCH ESO PRO REC1 PARAM26 VALUE = '25      ' / Default: 25
HIERARCH ESO PRO REC1 PARAM27 NAME = 'struct_refx' / Reference X value to comput
HIERARCH ESO PRO REC1 PARAM27 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM28 NAME = 'struct_refy' / Reference Y value to comput
HIERARCH ESO PRO REC1 PARAM28 VALUE = '-1      ' / Default: -1
HIERARCH ESO PRO REC1 PARAM29 NAME = 'pd_noise_compute' / Determine Fixed Patter
HIERARCH ESO PRO REC1 PARAM29 VALUE = 'false   ' / Default: false
HIERARCH ESO PRO REC1 PARAM30 NAME = 'pd_noise_dc_x' / x-size (pixel) of the mas
HIERARCH ESO PRO REC1 PARAM30 VALUE = '1       ' / Default: 1
HIERARCH ESO PRO REC1 PARAM31 NAME = 'pd_noise_dc_y' / y-size (pixel) of the mas
HIERARCH ESO PRO REC1 PARAM31 VALUE = '1       ' / Default: 1

I will use the same syntax to add:

raw frame input and categories
pipeline version number
calibration frames and categories
pipeline recipe parameters and values

thespacedoctor commented 1 year ago

A note from Andrea with much the same info as above:

Concerning what you asked we store information on input RAW files as:

HIERARCH ESO PRO REC1 RAW1 NAME = 'XSHOO.2010-05-19T04:40:49.818.fits' / File na
HIERARCH ESO PRO REC1 RAW1 CATG = 'OBJECT_SLIT_NOD_VIS' / Category of raw frame
HIERARCH ESO PRO REC1 RAW2 NAME = 'XSHOO.2010-05-19T04:42:28.806.fits' / File na
HIERARCH ESO PRO REC1 RAW2 CATG = 'OBJECT_SLIT_NOD_VIS' / Category of raw frame

...

Input calibrations:

HIERARCH ESO PRO REC1 CAL1 NAME = 'SPECTRAL_FORMAT_TAB_VIS.fits' / File name of
HIERARCH ESO PRO REC1 CAL1 CATG = 'SPECTRAL_FORMAT_TAB_VIS' / Category of calibr
HIERARCH ESO PRO REC1 CAL2 NAME = 'ARC_LINE_LIST_VIS.fits' / File name of calibr
HIERARCH ESO PRO REC1 CAL2 CATG = 'ARC_LINE_LIST_VIS' / Category of calibration
HIERARCH ESO PRO REC1 CAL3 NAME = 'MASTER_BIAS_VIS_1x2_100k_0.9x11.fits' / File
HIERARCH ESO PRO REC1 CAL3 CATG = 'MASTER_BIAS_VIS' / Category of calibration fr
HIERARCH ESO PRO REC1 CAL3 DATAMD5 = '3950e155fd95e5510b1505c34e888f07' / MD5 si
HIERARCH ESO PRO REC1 CAL4 NAME = 'MASTER_FLAT_SLIT_VIS_1x2_100k_0.9x11.fits'
HIERARCH ESO PRO REC1 CAL4 CATG = 'MASTER_FLAT_SLIT_VIS' / Category of calibrati
HIERARCH ESO PRO REC1 CAL4 DATAMD5 = 'abc4af497a79590ae3951dd6a4532945' / MD5 si
HIERARCH ESO PRO REC1 CAL5 NAME= 'ORDER_TAB_EDGES_SLIT_VIS_1x2_100k_0.9x11.fits'
HIERARCH ESO PRO REC1 CAL5 CATG = 'ORDER_TAB_EDGES_SLIT_VIS' / Category of calib
HIERARCH ESO PRO REC1 CAL5 DATAMD5 = '2b3b1d84ee4fd7861f7cabcaf850e511' / MD5 si
HIERARCH ESO PRO REC1 CAL6 NAME = 'xsh_atmoexan.fits' / File name of calibration
HIERARCH ESO PRO REC1 CAL6 CATG = 'ATMOS_EXT_VIS' / Category of calibration fram
HIERARCH ESO PRO REC1 CAL6 DATAMD5 = '6aea4595776baaa7efc244012254dd9c' / MD5 si
HIERARCH ESO PRO REC1 CAL7 NAME = 'XSH_MOD_CFG_OPT_2D_VIS_1x1_400k_Pin_row.fits'
HIERARCH ESO PRO REC1 CAL7 CATG = 'XSH_MOD_CFG_OPT_2D_VIS' / Category of calibra
HIERARCH ESO PRO REC1 CAL7 DATAMD5 = '646b365f432bde8c3b1fb6e344b08eec' / MD5 si

Note that the DATAMD5 is yseful to detect univocally in the ESO archive files of a given filename and PRO.CATG.

For input recipe parameters:

HIERARCH ESO PRO REC1 PARAM1 NAME = 'keep-temp' / If 'no', temporary files are d
HIERARCH ESO PRO REC1 PARAM1 VALUE = 'no      ' / Default: 'no'
HIERARCH ESO PRO REC1 PARAM2 NAME = 'debug-level' / Additional xshooter debug le
HIERARCH ESO PRO REC1 PARAM2 VALUE = 'none    ' / Default: 'none'
HIERARCH ESO PRO REC1 PARAM3 NAME = 'time-stamp' / Add timestamp to product file
HIERARCH ESO PRO REC1 PARAM3 VALUE = 'false   ' / Default: false
HIERARCH ESO PRO REC1 PARAM4 NAME = 'decode-bp' / Integer representation of the
HIERARCH ESO PRO REC1 PARAM4 VALUE = '1741684735' / Default: 1741684735
HIERARCH ESO PRO REC1 PARAM5 NAME = 'pre-overscan-corr' / pre-overscan correctio
HIERARCH ESO PRO REC1 PARAM5 VALUE = '1       ' / Default: 1
HIERARCH ESO PRO REC1 PARAM6 NAME = 'stack-method' / Method used to build master
HIERARCH ESO PRO REC1 PARAM6 VALUE = 'median  ' / Default: 'median'
HIERARCH ESO PRO REC1 PARAM7 NAME = 'klow    ' / Kappa used to clip low level va
HIERARCH ESO PRO REC1 PARAM7 VALUE = '5       ' / Default: 5

thespacedoctor / soxspipe

SOX-062-A08: Record input filenames in product FITS headers #138