desi_proc calls desi_compute_psf to calculate the PSF per exposure
desi_compute_psf (scripts.specex.main) calls desi_fit_psf to fit individual bundles, and then merges them back together with desi_merge_psf.
desi_proc_joint_fit then calls scripts.specex.mean_psf to combine PSFs across exposures into the nightly PSF
However, if desi_fit_psf fails for an individual bundle by having a bogus solution with overlapping fiber traces, this causes desi_compute_psf to fail and halt the entire pipeline.
It would be better if the bad bundle would be flagged as bad in the per-exposure PSF fit to prevent it from being used in mean_psf, but otherwise let the pipeline proceed. It is relatively common for small updates to darks/masks to trigger new edge case failures leading to time consuming flagging, cleanup, and resubmissions in new prods. Our current mitigation is to flag the entire camera as bad for that exposure and rerun, which looses the otherwise good bundles on that camera from the mean (ok, but not great).
Example failures of individual bundles in jura which had not been caught and flagged by earlier prods:
night 20221209 exp 157081 camera z4 fibers 225-250
night 20210327 exp 82431 camera r1 fibers 75-100
night 20220506 exp 133476 camera r2 fibers 275-300
All of these would have been fine if it just logged an error, flagged the per-exposure bundle as bad, but then proceeded and dropped the bundle/exp from the psfnight merging.
Currently in the pipeline
desi_proccallsdesi_compute_psfto calculate the PSF per exposuredesi_compute_psf(scripts.specex.main) callsdesi_fit_psfto fit individual bundles, and then merges them back together withdesi_merge_psf.desi_proc_joint_fitthen callsscripts.specex.mean_psfto combine PSFs across exposures into the nightly PSFHowever, if
desi_fit_psffails for an individual bundle by having a bogus solution with overlapping fiber traces, this causesdesi_compute_psfto fail and halt the entire pipeline.It would be better if the bad bundle would be flagged as bad in the per-exposure PSF fit to prevent it from being used in
mean_psf, but otherwise let the pipeline proceed. It is relatively common for small updates to darks/masks to trigger new edge case failures leading to time consuming flagging, cleanup, and resubmissions in new prods. Our current mitigation is to flag the entire camera as bad for that exposure and rerun, which looses the otherwise good bundles on that camera from the mean (ok, but not great).Example failures of individual bundles in jura which had not been caught and flagged by earlier prods:
All of these would have been fine if it just logged an error, flagged the per-exposure bundle as bad, but then proceeded and dropped the bundle/exp from the psfnight merging.