PSF fitting fails on arc images when one amp is bad, e.g. 20230428 when b8B was bad all night:
(UPDATE: added missing --badamps b8B option to example)
srun -n 21 -c 8 desi_proc --mpi -n 20230428 -e 177975 --cameras b8 --badamps b8B
...
ERROR:qa.py:37:trace_psf_qa: overlapping traces for fibers 318 and 319 in /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposures/20230428/00177975/fit-psf-b8-00177975_12.fits
ERROR:specex.py:199:main: desi_psf_fit on process 12 failed with return value 1 running desi_psf_fit -a /dvs_ro/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/preproc/20230428/00177975/preproc-b8-00177975.fits.gz --in-psf /dvs_ro/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposures/20230428/00177975/shifted-input-psf-b8-00177975.fits --out-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposures/20230428/00177975/fit-psf-b8-00177975_12.fits --lamp-lines /global/common/software/desi/perlmutter/desiconda/20230111-2.1.0/code/specex/main/py/specex/data/specex_linelist_desi.txt --first-bundle 12 --last-bundle 12 --first-fiber 300 --last-fiber 324 --legendre-deg-wave 1 --broken-fibers 348,473,474
...
ERROR:specex.py:340:fitframe: FAILED: MPI ranks 1-20 on camera b8
ERROR:specex.py:342:fitframe: FAILED: desi_compute_psf --input-image /dvs_ro/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/preproc/20230428/00177975/preproc-b8-00177975.fits.gz --input-psf /dvs_ro/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposures/20230428/00177975/shifted-input-psf-b8-00177975.fits --output-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/procnight/exposures/20230428/00177975/fit-psf-b8-00177975.fits --broken-fibers 348,473,474
ERROR:specex.py:343:fitframe: some bundles failed desi_psf_fit
INFO:specex.py:347:fitframe: specex fit for b8 took 15.6 seconds
...
We have 26 nights impacted by this problem. Previously we discarded the entire camera for cals if an amp was bad, but this is non-ideal because it then falls back to a default calibration which is old and possibly inconsistent with the cameras that were processed.
The wrapper knows that the bundles failed, but needs to be updated to write output anyway with the bad bundles flagged as bad. That's ok for these nights where the amp is bad all night, because we don't need that bundle anyway (as long as it also doesn't mess up fiberflat normalization due to bad extractions on those bundles, TBD).
Also: sometimes individual bundles on individual exposures also fails. When this happens in dailyops we flag that arc camera as bad and move on, letting the PSF night be combined from 4 instead of 5 exposures. If the PSF fitting is updated such that individual failed bundles is no longer fatal for the exposure, we need to make sure that doesn't propagate into a bad combined fit when that exposure is combined with others. I'll post an example of that when I find one; spot checking ones that previously failed they are now working...
sbailey
commented
2 months ago
PSF fitting fails on arc images when one amp is bad, e.g. 20230428 when b8B was bad all night: (UPDATE: added missing
--badamps b8Boption to example)We have 26 nights impacted by this problem. Previously we discarded the entire camera for cals if an amp was bad, but this is non-ideal because it then falls back to a default calibration which is old and possibly inconsistent with the cameras that were processed.
The wrapper knows that the bundles failed, but needs to be updated to write output anyway with the bad bundles flagged as bad. That's ok for these nights where the amp is bad all night, because we don't need that bundle anyway (as long as it also doesn't mess up fiberflat normalization due to bad extractions on those bundles, TBD).
Also: sometimes individual bundles on individual exposures also fails. When this happens in dailyops we flag that arc camera as bad and move on, letting the PSF night be combined from 4 instead of 5 exposures. If the PSF fitting is updated such that individual failed bundles is no longer fatal for the exposure, we need to make sure that doesn't propagate into a bad combined fit when that exposure is combined with others. I'll post an example of that when I find one; spot checking ones that previously failed they are now working...