Open akremin opened 3 years ago
sbailey
commented
3 years ago Is this discrepancy just confusing, or are we actually using FILENAME downstream to do something?
We could consider intercepting and correcting or dropping this keyword as a special case in desi_preproc. Spot checking data from 20210615, it appears that keyword no longer exists.
akremin
commented
3 years ago I don't know of any downstream pipeline issues with it, but it is being propagated to our data products (I found it via looking at the fiberflats). So those files no longer document the raw data they were derived from, and point to files that don't exist.
I was using this to find the raw data associated with each derived file to find additional metadata that wasn't being propagated into the fibermap or header. Because of the issue, I instead assume the raw data used was in the nominal $DESI_SPECTRO_DATA, which is fine for my purposes.
I now see that you are correct, the keyword was dropped in later raw data. However, it is re-derived somewhere in the pipeline because the data products on those later nights still have it.
Frames for June 15, 2021:
kremin@cori07:/global/cfs/cdirs/desi/spectro/redux/everest/exposures> fitsheader -e 0 -k "FILENAME" ./20210615/*/frame*b1*fits
# HDU 0 in ./20210615/00094674/frame-b1-00094674.fits:
FILENAME= '/exposures/desi/20210615/00094674/desi-00094674.fits.fz'
# HDU 0 in ./20210615/00094675/frame-b1-00094675.fits:
FILENAME= '/exposures/desi/20210615/00094675/desi-00094675.fits.fz'
# HDU 0 in ./20210615/00094676/frame-b1-00094676.fits:
FILENAME= '/exposures/desi/20210615/00094676/desi-00094676.fits.fz'
# HDU 0 in ./20210615/00094677/frame-b1-00094677.fits:
FILENAME= '/exposures/desi/20210615/00094677/desi-00094677.fits.fz'
# HDU 0 in ./20210615/00094678/frame-b1-00094678.fits:
FILENAME= '/exposures/desi/20210615/00094678/desi-00094678.fits.fz'
# HDU 0 in ./20210615/00094679/frame-b1-00094679.fits:
FILENAME= '/exposures/desi/20210615/00094679/desi-00094679.fits.fz'
# HDU 0 in ./20210615/00094680/frame-b1-00094680.fits:
FILENAME= '/exposures/desi/20210615/00094680/desi-00094680.fits.fz'
# HDU 0 in ./20210615/00094681/frame-b1-00094681.fits:
FILENAME= '/exposures/desi/20210615/00094681/desi-00094681.fits.fz'
# HDU 0 in ./20210615/00094682/frame-b1-00094682.fits:
FILENAME= '/exposures/desi/20210615/00094682/desi-00094682.fits.fz'
# HDU 0 in ./20210615/00094683/frame-b1-00094683.fits:
FILENAME= '/exposures/desi/20210615/00094683/desi-00094683.fits.fz'
# HDU 0 in ./20210615/00094684/frame-b1-00094684.fits:
FILENAME= '/exposures/desi/20210615/00094684/desi-00094684.fits.fz'
# HDU 0 in ./20210615/00094685/frame-b1-00094685.fits:
FILENAME= '/exposures/desi/20210615/00094685/desi-00094685.fits.fz'
# HDU 0 in ./20210615/00094686/frame-b1-00094686.fits:
FILENAME= '/exposures/desi/20210615/00094686/desi-00094686.fits.fz'
# HDU 0 in ./20210615/00094687/frame-b1-00094687.fits:
FILENAME= '/exposures/desi/20210615/00094687/desi-00094687.fits.fz'
# HDU 0 in ./20210615/00094688/frame-b1-00094688.fits:
FILENAME= '/exposures/desi/20210615/00094688/desi-00094688.fits.fz'
# HDU 0 in ./20210615/00094689/frame-b1-00094689.fits:
FILENAME= '/exposures/desi/20210615/00094689/desi-00094689.fits.fz'
# HDU 0 in ./20210615/00094690/frame-b1-00094690.fits:
FILENAME= '/exposures/desi/20210615/00094690/desi-00094690.fits.fz'
# HDU 0 in ./20210615/00094691/frame-b1-00094691.fits:
FILENAME= '/exposures/desi/20210615/00094691/desi-00094691.fits.fz'
.... (truncated)For reference, Dec 2020 frames where the issue still presents itself:
kremin@cori07:/global/cfs/cdirs/desi/spectro/redux/everest/exposures> fitsheader -e 0 -k "FILENAME" ./20201214/*/frame*b1*fits
# HDU 0 in ./20201214/00067628/frame-b1-00067628.fits:
FILENAME= '/exposures/desi/specs/20201214/00067628/sp8-00067628.fits.fz' / Name
# HDU 0 in ./20201214/00067629/frame-b1-00067629.fits:
FILENAME= '/exposures/desi/specs/20201214/00067629/sp7-00067629.fits.fz' / Name
# HDU 0 in ./20201214/00067630/frame-b1-00067630.fits:
FILENAME= '/exposures/desi/specs/20201214/00067630/sp6-00067630.fits.fz' / Name
# HDU 0 in ./20201214/00067633/frame-b1-00067633.fits:
FILENAME= '/exposures/desi/specs/20201214/00067633/sp4-00067633.fits.fz' / Name
# HDU 0 in ./20201214/00067634/frame-b1-00067634.fits:
FILENAME= '/exposures/desi/specs/20201214/00067634/sp1-00067634.fits.fz' / Name
# HDU 0 in ./20201214/00067635/frame-b1-00067635.fits:
FILENAME= '/exposures/desi/specs/20201214/00067635/sp0-00067635.fits.fz' / Name
# HDU 0 in ./20201214/00067638/frame-b1-00067638.fits:
FILENAME= '/exposures/desi/specs/20201214/00067638/sp6-00067638.fits.fz' / Name
# HDU 0 in ./20201214/00067639/frame-b1-00067639.fits:
FILENAME= '/exposures/desi/specs/20201214/00067639/sp8-00067639.fits.fz' / Name
# HDU 0 in ./20201214/00067640/frame-b1-00067640.fits:
FILENAME= '/exposures/desi/specs/20201214/00067640/sp8-00067640.fits.fz' / Name
# HDU 0 in ./20201214/00067643/frame-b1-00067643.fits:
FILENAME= '/exposures/desi/specs/20201214/00067643/sp6-00067643.fits.fz' / Name
# HDU 0 in ./20201214/00067644/frame-b1-00067644.fits:
FILENAME= '/exposures/desi/specs/20201214/00067644/sp4-00067644.fits.fz' / Name
# HDU 0 in ./20201214/00067645/frame-b1-00067645.fits:
FILENAME= '/exposures/desi/specs/20201214/00067645/sp7-00067645.fits.fz' / Name
.... (truncated)
In tracing back an issue with the keyword "FILENAME" in fiberflats I found that it stemmed from the raw data headers themselves.
For data up to and including 20210104, the "FILENAME" keyword doesn't follow the
desi-<EXPID>.fits.fztemplate, and additionally the spectrograph listed in the file names are also discrepant with the actual spectrograph (SP) value and appear to be random.Note below I'm looking at the "B1" headers. Spot checking other camera-spec combinations showed the same behavior.
Examples: