robustness to DARKs that span noon NIGHT rollover

[sbailey]

DARK exposure 20211205/00112406/desi-00112406.fits.fz is under NIGHT 20211205 but has header NIGHT=20211204. Unluckily, it was also a 300s dark, which appears to have taken the spot as the first 300 sec DARK of the night even though the header keyword mismatch prevented it from getting used (initially no dark*.slurm job was created; I submitted the "correct" dark 112463 by hand).

Various remediations to be more robust:

• only use DARK if PROGRAM starts with "CALIB Dark" (from the zeros+dark sequence) instead of "DARK" from the 90min dark sequences.
• if an exposure is rejected, don't let it take the spot for the nightly badcolumn dark

[akremin]

This is a useful discussion to have and come to a definitive answer on using PROGRAM for exposure filtering. Given that these PROGRAM values are pre-defined by the scripts used to run them they should be as robust as any other parameter, but we have had a collective hesitation because there is no precise reason for a given name and a well meaning person could change it to something else without realize the hard coded values in the code.

All of our cals have multiple subtypes. We currently use EXPTIME information to distinguish most of them and SEQTOT to distinguish between flat sequence types. We could do belt-and-suspender by adding PROGRAM checks, or alternatively move to a PROGRAM check rather than arbitrary exposure times.

Summary of types of cal follow:

zeros

1) "flush" zeros after turning VCCD on. 2) Actual bias frames: b camera nightly biases and generating bias model

darks

1) 300s calibration dark: for bad column detection and dark model evolution modeling. 2) Dark sequence(s): for dark modeling

arcs

1) short arcs: for psf modeling 2) long arcs: for faint lines, not used currently

flats

1) short flat sequence: no longer used 2) long flat sequence: for flat fielding 3) short flat exposure: for cte identification. There are three cases where we have multiple sub-types of obstypes: short vs long arcs, short flat sequence

[sbailey]

This happened again with a dark 20241124/00264676/desi-00264676.fits.fz that spanned the noontime boundary and ended up with NIGHT=20241123 in the headers. This led to pipeline bookkeeping confusion such that the 20241124 processing re-ran the 20241123 ccdcalib and then crashed because calibnight/20241124 files hadn't been created.

At minimum we could fail faster and more informatively in this case. Or better the exposure table creation could recognize the inconsistency and auto-flag the exposure as bad. Or it would probably work to always believe the directory as true and override the header keyword for downstream propagation if they disagree.