check DESI_SPECTRO_DARK for missing darks

[sbailey]

@Waelthus in preparation for our next spectro run, please check that /global/cfs/cdirs/desi/spectro/desi_spectro_dark/v2209/ has default darks and biases for all cameras for all nights since 20201214. e.g. something like:

Loop over nights, read one exposure per night, run calibfinder to get default dark and bias, and look for messages like

ERROR:calibfinder.py:546:find_darks_in_desi_spectro_dark: Didn't find matching r5 calibration darks in $DESI_SPECTRO_DARK using default from $DESI_SPECTRO_CALIB instead

It looks like there are early dates where the default darks haven't been copied from DESI_SPECTRO_CALIB -> DESI_SPECTRO_DARK, but there are also more recent cases of missing darks, e.g.

$> echo $DESI_SPECTRO_DARK
/global/cfs/cdirs/desi/spectro/desi_spectro_dark/v2209
$> desi_preproc -n 20231111 -e 204371 --cameras r5
...
ERROR:calibfinder.py:546:find_darks_in_desi_spectro_dark: Didn't find matching r5 calibration darks in $DESI_SPECTRO_DARK using default from $DESI_SPECTRO_CALIB instead
...

This is related to PR #2162 where the new default will be to fail instead of falling back. Let's find and fix those missing darks ahead of time.

[Waelthus]

Ok, I partially tackled this problem and here's a summary of the current state:

list of nights that were fixed (as in reduction runs, did not validate results yet) by now:	Camera	Nights	Cause of issue	Fix
all but b0	20201214 - 20211229	no DARK before 20201229 in DESI_SPECTRO_DARK	symlink to existing 20201229 dark
b0	20201214 - 20210129	dark observations in early Dec 2021 had b0 in BADCAMWORDs => no 20201229 dark possible	generate dark model from late Dec 20 and Jan 21
b5	20210504-20210621	change of CCDCFG wrt before	new dark based on 20210601 and 20210616 observations
r9	20210523-20210609	CCDTEMP decreased from -139.67 to -142.33 C within the timeframe (with the model being at 137.61), not enough DARKs taken for new model (but some are taken, just not 5 per EXPTIME)	computed a dark model with less darks/EXPTIME, gets accepted for 20210523-20210609, need to check performance
r9	20210615-20210628	change of CCDCFG	symlink to existing 20210630 dark
r9	20210610	same as 20210523-20210609, but this is just outside the temperature threshold of the new dark model	could slightly increase threshold to 2.3K or similar
b5	20210625-20210628	new CCDCFG after cryostat change	symlink to existing 20210630 dark
z5	20210625-20210628	CCDTEMP showing -200.0 C for that timeframe (maybe temp data is actually missing as the value is exact), not enough DARKs taken for new model	use copy of dark model from 20210630, change header to CCDTEMP of -200K
r1	20211015-20211024	new CCDCFG	symlink to existing 20211025 darks
r4	20211015-20211024	new CCDCFG	symlink to existing 20211025 darks
r8	20211019-20211024	new CCDCFG	symlink to existing 20211025 darks
r8	20211117-20211118	CCDTMING of lbnl_timing2left.txt; actually this should've failed starting 20211104, not sure why it didn't there	generated full dark model starting on 20211104, but part of the frames had 3h<VCCDON<6h	20211118
r8	20211119-20211214	not actually failing, but an inconsistency in CCDTMING between bias and dark model, currently falling back to 202110 darks	redo the December dark model excluding the 20211118 frames, let it start on 20211119
z3	20211202-20211213	CCDTMING	symlink to existing 20211214 darks
r1	20220104-20220113	new CCDCFG	symlink to existing 20220114 darks
z1	20220104-20220106	different CCDCFG for just 2 days	use same dark model as for nights after, change CCDCFG entry in the model manually, needs validation
z1	20220107-20220113	new CCDCFG	symlink to existing 20220114 darks
z1	20220602-20220612	new CCDCFG	symlink to existing 20220613 dark
z3	20220531-20220613	new CCDCFG starting 20220531, but dark models from that date and from 20220613 were still based on the old ones (at least according to their CCDCFG header)	recreate dark model based on June 2022 observations
z3	20220906-20220912	same as above	symlink to existing 20220913 darks (should probably do so also for other cameras, so the post-shutdown darks will be used everywhere from the restart)
b4,r4	20221109-20221126	DETECTOR	symlink to existing 20221127 dark
z5	20221109-20221114	different DETECTOR, CCDCFG not matching time after	generating new dark with exps taken in this timeframe only
z5	20221115-20221124	different DETECTOR than before	link to the 20221128 dark
r7	20230323-20230402	CCDCFG changed, but not enough DARKs for new model	symlink to 20230403 darks
r4	20230427-20230507	CCDTEMP for data at -158 C, but closest model at -155 C, not enough darks taken for new model	symlink to 20230508 darks
z1	20230411-20230501	CCDTEMP for data at -144 C, but closest model at -134 C, not enough darks taken for new model	generating linear dark model from all the 1200s darks
b1,r4	20230805-20230814	new CCDCFG	symlink to existing 20230815 darks (did this for all cameras)
z1	20230822-20230824	different CCDCFG for just 3 days, have 3 sets of dark exposures	computed dark model on those darks, needs validation
r5	20231003-20231125	CCDTEMP stored 850 C (I think this is a default value for missing data, B-CCDs store this throughout) for this timeframe	ran new darks using data until 20231028 for all CCDs (an additional dark for Oct 23 was missing anyway), linked to that for r5, dark creation failed for sm5/6 which were not used on 20231028

nights where fixing isn't done yet:	Camera	Nights	Cause of issue	Proposed Fix	Science exp taken?

nights with data mentioned in the exposure table, but no data in DESI_SPECTRO_DATA (incomplete list, only noted the ones I accidentally found, I guess in those nights the whole night might be missing since both exps at the beginning and end are not there):	Nights	EXPID
20210628	96521,96670
20211104	107440,107480
20211108	108049,108120
20211115	108976,109133
20211203	112196,112346
20211204	112406,112463
20220112	118415,118443
20221229	161211,161400
20230510	180072,180204
20230921	197450,197706
20231212	209383,209652
20231213	209664,209865

Cases with multiple setups within exposure tables:	Camera	Night	EXPID setup 1 (working)	EXPID setup 2 (not working)
b0	20201216	68214	68168
z6	20211107	107785	107735
z3	20211202	111798	111828
z1	20230824	192401	192230

[sbailey]

@Waelthus thanks for this detailed list. Suggested next steps:

• We only care about configurations (DETECTOR, CCDCFG, CCDTMING, CCDTEMP) for which we actually took science data. Please check if any of the "fixing isn't so easy" cases are just about non-science data and thus could be ignored.
• z5 | 20210625-20210628 CCDTEMP=-200 that sounds like a sensor/readout failure. Please check the configs immediately before and after that and compare to the 300s darks during that time period. It is probably safe to use one of the before/after darks as the model during this time.
• Idea from Julien: for the remaining cases, we could make a normal dark using the 1200 second morning darks, and test them on the 300 sec evening darks (or even use some of those 300s evening darks while making the model if needed). These wouldn't be the full non-linear dark models, but they are probably good enough for most science exposures which are mostly longer than the non-linear regime. I'm not sure if this is "desi_make_regular_darks" or "desi_compute_dark".

Context/History for the record: typically a CCD dark current model scales linearly with time, i.e. you can make a model using 1200s exposures and then apply it to a 900s exposure just by multiplying by (900/1200). Unfortunately the DESI CCDs have a non-linear dark at the edges of the CCD, which primarily impacts exposures <60s which is shorter than most of our science exposures. See DESI-5761 for the original analysis, with timing file improvements from Julien and Armin on 20210201 significantly reducing the size of the effect, see DESI-6079. With those timing file improvements, the effect is just barely big enough to matter on a few CCDs, but it isn't a disaster if we have to make regular linear dark models using just long darks without having the full A-E sequences that include short darks to be able to build the time-dependent model.

[Waelthus]

Ok, just a little update on this. All of the failure modes have had science observations in the timeframe, and assuming a fix for the timeframe works throughout the timeframe will work on fixing these now.

For r7 on 20230323-20230402 a fix was easily doable after all, by symlinking the next dark from 20230403 (which I seem to have not tried last week). For r4 issues in Apr/May 2023, the same is true.

For the issues where CCDTEMP has not been correctly stored, the easiest solution would be to use a copy of a close-by dark model and change its CCDTEMP to an accepted value. This would have the advantage of only being used in that config, but the disadvantage of having values in the model FITS files that aren't actually correct. Similarly, for z1 on 20220104-20220106, where we did not take any darks except the 300s, but maybe the darks from 20220107 with slightly different config are good enough? Not sure if a linear dark from the 3 nightly 300s exposures would be good enough. For the other CCDTEMP cases it was possible to compute new darks (linear for z1 on 20230411-20230501 because we only had 1200s frames in the timespan, nonlinear for r4 on 20230427-20230507 and nonlinear with a reduced number of darks per exptime for r9 on 20210523-20210609, r9 on 20210610 passes the temperature threshold by 0.3K, so I suggest increasing it from 2K to 2.5K to allow this as well) For z1 on 20230822-20230824, we could try producing a non-linear dark out of the 3 sequences that we have, I guess, or switch to just linear darks. For z5 on 20221109-20221124 I figured out that the actual change date was 20221119, we took enough darks both before and after, but the config was not being picked up correctly before because of an inconsistency in the table, for later data just a fallback to 202210 darks happened, now this is done correctly.

[Waelthus]

@sbailey: for every night now dark models are picked up again. Running one nightly dark and one morning dark for each night in the whole 3 years of data through preproc now to double-check and to see how things evolve.

[sbailey]

@Waelthus thanks for the updates. I still have dark failures for the following night,expid,camera combinations:

import fitsio
import desispec.io
from desispec.calibfinder import CalibFinder

for night, expid, camera in [
    (20210610, 93286, 'R9'),
    (20211109, 108049, 'Z6'),
    (20211118, 109266, 'Z6'),
    (20211118, 109410, 'Z6'),
    (20211202, 111907, 'Z3'),
    (20211115, 108931, 'Z6'),
    (20211116, 108976, 'Z6'),
    ]:
    rawfile = desispec.io.findfile('raw', night, expid)
    hdr = fitsio.read_header(rawfile, 1)
    camhdr = fitsio.read_header(rawfile, camera)

    try:
        cf = CalibFinder([hdr, camhdr])
        print(f'{night} {expid} {camera} - WORKED')
    except IOError:
        print(f'{night} {expid} {camera} - FAILED')

Results:

CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching r9 calibration darks in $DESI_SPECTRO_DARK, quitting
20210610 93286 R9 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z6 calibration darks in $DESI_SPECTRO_DARK, quitting
20211109 108049 Z6 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z6 calibration darks in $DESI_SPECTRO_DARK, quitting
20211118 109266 Z6 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z6 calibration darks in $DESI_SPECTRO_DARK, quitting
20211118 109410 Z6 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z3 calibration darks in $DESI_SPECTRO_DARK, quitting
20211202 111907 Z3 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z6 calibration darks in $DESI_SPECTRO_DARK, quitting
20211115 108931 Z6 - FAILED
CRITICAL:calibfinder.py:555:find_darks_in_desi_spectro_dark: Didn't find matching z6 calibration darks in $DESI_SPECTRO_DARK, quitting
20211116 108976 Z6 - FAILED

Those exposures came from the production exposure_tables with entries with LASTSTEP='all'. If any of them are test exposures with wild settings that aren't supposed to have darks, let me know and we should flag them in the exposure tables as bad.

[Waelthus]

@sbailey Ok, the checks I did where typically on the nightly DARK exposure (falling back to the first and last exposure of the night where we didn't have that) with my assumption being that if that works the whole night should work. The exposure tables I used there where the ones in $DESI_SPECTRO_REDUX/daily/exposure_tables/ and I do find the exposures you mention in there. So my main assumption of configs not changing within the night seems to be wrong and I should potentially loop through every exposure taken (and previously reduced acc to LASTSTEP) instead. For the entries you quote I find:

• 20210610 works until (and incl) 93271, but the detector became warmer throughout the night, so my 2.5K threshold is too low to capture the amount of change (2.3K difference to the closest DARK at the beginning of the night 2.73K at the end of the NIGHT, there was a steady change in temperature totalling ~3-3.5K over the 2 weeks before), note that the science exposures of that NIGHT are marked with EXPFLAG==['aborted'] in the exposure tables (not fully sure what that signifies)... The easiest change would be to allow a slightly larger temperature range which I now did.
• 20211109: there were DARKs and spectests done during the day using 2amp down mode, then the instrument was changed back to the previous config (starting exp 108080 with calib zeros and calib dark) which was used throughout the rest of the night
• 20211202: we took two calib darks, I only checked the first one, but obs were performed with the setting of the second one (which was also used on subsequent days) -> enabled the new dark one day earlier and now gets picked up
• 20211115, exp 108902 and beyond fail (only ZERO/DARK), but after 108976 a BADCAMWORD of a6 and ['z6 2amp mode bad' 'z6 takes down all of p6'] in the comments is set for multiple days (until incl 20211118), so didn't try to really address this

If there are any more files where we know of failures (or have an easier way to find them then actually going through every single exp in the lists), I'd happily track them down. But I think for most of these cases the issue was just a config change somewhere in the night.

[sbailey]

Thanks! Summarizing:

• 20210610 and 20211202 are now working due to updated darks and temp thresholds
• I flagged the z6 2amp mode test data in November 2021 as bad so my checks no longer trip on those as not having darks.

All cameras on all nights with good data now pass the $DESI_SPECTRO_DARK check. Success!