20231028 camera z3 amplifier C: needs CTE correction or not?

[sbailey]

Blocking factor for processing Jura night 20231028

The jura prod ran CTE corrections with:

desi_fit_cte_night -n 20231028 -c a01456789 -e 202529,202530

This generated calibnight/20231028/ctecorr-20231028.yaml with CTE corrections for z1C and z7B.

However, all arc jobs failed while preprocessing z3 with error messages like:

INFO:preproc.py:1385:preproc: Applying CTE correction for camera z3
CRITICAL:correct_cte.py:334:get_cte_params: No CTE correction in /dvs_ro/cfs/cdirs/desi/spectro/redux/jura/calibnight/20231028/ctecorr-20231028.yaml for night 20231028 camera z3 amplifier C
CRITICAL:util.py:155:runcmd: FAILED rank 0 exception while running desispec.scripts.preproc.main(['-i', '/dvs_ro/cfs/cdirs/desi/spectro/data/20231028/00202496/desi-00202496.fits.fz', '-o', '/global/cfs/cdirs/desi/spectro/redux/jura/preproc/20231028/00202496/preproc-z3-00202496.fits.gz', '--outdir', '/global/cfs/cdirs/desi/spectro/redux/jura/preproc/20231028/00202496', '--cameras', 'z3']) called from line 444 in /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/scripts/proc.py
ERROR:util.py:163:runcmd: Traceback (most recent call last):
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/util.py", line 146, in runcmd
    result = cmd(*args)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/scripts/preproc.py", line 231, in main
    num_failed += _preproc_file_kwargs_wrapper(opts)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/scripts/preproc.py", line 249, in _preproc_file_kwargs_wrapper
    return preproc_file(**opts)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/scripts/preproc.py", line 274, in preproc_file
    img = io.read_raw(infile, camera,
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/io/raw.py", line 185, in read_raw
    img = desispec.preproc.preproc(rawimage, header, primary_header, **kwargs)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/preproc.py", line 1386, in preproc
    img = desispec.correct_cte.correct_image_via_model(img,niter=5,cte_params_filename=cte_params_filename)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/correct_cte.py", line 980, in correct_image_via_model
    amp, cte = get_cte_params(image.meta, cte_params_filename=cte_params_filename)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.1/lib/python3.10/site-packages/desispec/correct_cte.py", line 335, in get_cte_params
    raise RuntimeError(mess)
ERROR:util.py:163:runcmd: RuntimeError: No CTE correction in /dvs_ro/cfs/cdirs/desi/spectro/redux/jura/calibnight/20231028/ctecorr-20231028.yaml for night 20231028 camera z3 amplifier C
CRITICAL:util.py:172:runcmd: FAILED desispec.scripts.preproc.main(['-i', '/dvs_ro/cfs/cdirs/desi/spectro/data/20231028/00202496/desi-00202496.fits.fz', '-o', '/global/cfs/cdirs/desi/spectro/redux/jura/preproc/20231028/00202496/preproc-z3-00202496.fits.gz', '--outdir', '/global/cfs/cdirs/desi/spectro/redux/jura/preproc/20231028/00202496', '--cameras', 'z3'])

@schlafly can you track down whether 20231028 z3C needs a CTE correction or not, and why desi_fit_cte_night and desi_preproc seem to disagree about that?

[schlafly]

I should look more, but I suspect that this is the issue

desi_fit_cte_night -n 20231028 -c a01456789 -e 202529,202530

which I take to mean that you have explicitly requested that desi_fit_cte_night not compute CTE corrections for petals 2 and 3. This then leads to problems applying CTE corrections to petal 3 (petal 2 doesn't have any CTE).

We do have offcols in DESI_SPECTRO_CALIB for that night, so we do expect preproc to want to find corrections and desi_fit_cte_night to compute them, unless it is explicitly told not to compute them.

[sbailey]

@schlafly thanks for reading the command correctly and noticing that the long camword string was missing "2" and "3". It is then an upstream problem for why the pipeline didn't ask for CTE corrections for those cameras even though it did ask it for other cameras that don't need a correction. I think the intension was for the pipeline to give desi_fit_cte_night the superset of cameras on the night and let desi_fit_cte_night pick which ones actually need a correction, but perhaps we have a union vs. intersection bug in camera bookkeeping.

[schlafly]

I did look briefly at that night and I suspect it's a case where something like the P2 and P3 are problematic at the beginning of the night and were marked to be masked. But then later in the night we no longer mask them. If that's right there's some mess about needing to get some cals from earlier nights, etc..

[schlafly]

But I'm going to take your comment as absolving me of responsibility for this particular issue---at least for the moment---can you assign someone else? :-)