exposure_tables files to specify calib overrides

[sbailey]

Add exposure_tables/YEARMM/ files and workflow support to specify manual overrides of how calibrations are processed, e.g.

• link to specific different night for calibnight files, to be used if no good cals were taken on a night
• extra options to autocalib_fiberflat (see #2172 --fit-gradient option)
• etc

Common cases should be handled by the normal exposure_tables/YEARMM/exposure_table_YEARMMDD.csv entries + workflow logic, but these additional files + workflow could handle cases where humans need to make decisions (which other night should be used if cals are bad) or if the analysis to determine the right thing to do is too complex for the workflow (should autocalib_fiberflat fit a gradient or not for NIGHT).

[sbailey]

PR #2187 added support for override files. Pulling comments from that long PR for override files that we need:

NIGHT	ZERO	DARK	ARC	FLAT	Comment
20211128	no	no	ok	ok	A: link bias,badcol but not psf,fiberflat
20220923	ok	ok	no	no	B: link psf,fiberflat, but not bias,badcol
20221222	ok	ok	no	no	B: link psf,fiberflat, but not bias,badcol
20221228	ok	ok	no	no	B: link psf,fiberflat, but not bias,badcol
20231216	ok	ok	no	no	B: link psf,fiberflat, but not bias,badcol
20210130	ok	ok	ok	no	C: link fiberflat, but not bias,badcol,psf
20210218	ok	ok	ok	no	C: link fiberflat, but not bias,badcol,psf
20211009	ok	ok	ok	no	C: link fiberflat, but not bias,badcol,psf
20221120	ok	ok	ok	no	C: link fiberflat, but not bias,badcol,psf
20231018	ok	ok	ok	no	C: link fiberflat, but not bias,badcol,psf
20211212	ok	ok	no	no	D: cals exist but with some arc lamps off, so link psf,fiberflat, but not bias,badcol,psf

I have implemented these in /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposure_tables:

202101/override_20210130.yaml
202102/override_20210218.yaml
202110/override_20211009.yaml
202111/override_20211128.yaml
202112/override_20211212.yaml
202209/override_20220923.yaml
202211/override_20221120.yaml
202212/override_20221222.yaml
202212/override_20221228.yaml
202310/override_20231018.yaml
202312/override_20231216.yaml

@akremin please review those, and if they are ok I can commit to svn and we can close this ticket.

[akremin]

For all of the cases of bad flats after 20211130, can we trust the 1s cte flat and last 120s flat for cte corrections? I don't see ctecorrnight listed in any of the override files, so I wanted to verify.

[sbailey]

Good catch. I reviewed these cases and there are some nights where we could do exposure table surgery to trick it into using 2 flats for CTE while not using them for fiberflats, but I think it is easier / safer / good-enough to just also link the ctecorr for any night after 20211130 that we also link fiberflats. I'll update the override files with that.

[akremin]

Sounds good. And as a friendly reminder we use the full name "ctecorrnight" for the override files

[sbailey]

I have added these override files for linkcalib cases:

Adding         202101/override_20210130.yaml
Adding         202102/override_20210218.yaml
Adding         202110/override_20211009.yaml
Adding         202111/override_20211128.yaml
Adding         202112/override_20211212.yaml
Adding         202209/override_20220923.yaml
Adding         202211/override_20221120.yaml
Adding         202212/override_20221222.yaml
Adding         202212/override_20221228.yaml
Adding         202310/override_20231015.yaml
Adding         202310/override_20231018.yaml
Adding         202312/override_20231216.yaml

@akremin please add the files for the fiberflatnight gradient overrides, and then I think we can close this ticket.

[akremin]

The following were added to the SVN. Note that one of the autocalib_fiberflat override files was submitted above, but was replaced with the one generated from the new override script generator. A second file submitted above for linkcal also required autocalib_fiberflat flags, so that one is appended and now includes information needed by both functions:

A    202401/override_20240101.yaml
A    202401/override_20240102.yaml
A    202309/override_20230925.yaml
A    202310/override_20231001.yaml
A    202310/override_20231011.yaml
A    202310/override_20231003.yaml
A    202310/override_20231013.yaml
A    202310/override_20231005.yaml
U    202310/override_20231015.yaml
A    202310/override_20231007.yaml
A    202310/override_20231009.yaml
A    202310/override_20231010.yaml
A    202310/override_20231002.yaml
A    202310/override_20231012.yaml
A    202310/override_20231004.yaml
A    202310/override_20231014.yaml
A    202310/override_20231006.yaml
A    202310/override_20231008.yaml
A    202312/override_20231203.yaml
A    202312/override_20231221.yaml
A    202312/override_20231231.yaml
A    202312/override_20231205.yaml
A    202312/override_20231207.yaml
A    202312/override_20231217.yaml
A    202312/override_20231219.yaml
A    202312/override_20231229.yaml
A    202312/override_20231220.yaml
A    202312/override_20231230.yaml
A    202312/override_20231204.yaml
A    202312/override_20231214.yaml
U    202312/override_20231216.yaml
A    202312/override_20231218.yaml
A    202312/override_20231228.yaml

[akremin]

All known issues requiring overrides are now represented by files in SVN. Closing this ticket.