search

desihub / specex

DESI spectrograph PSF fitting
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

b5 PSF fits failing May 2021 #59

Closed sbailey closed 2 years ago

sbailey commented 2 years ago

b5 PSF fits are regularly failing for May 2021 data, e.g. on nights 20210507, 0508, and 0509 every b5 arc exposures fails, thus losing the spectrograph for the night unless we do by-hand fallback to other calibs.

Curiously, these did not fail for the Everest production. Known changes since then: everest test-21.12
desi_spectro_calib 0.2.7 trunk
specex 0.8.0 0.8.3
desispec 0.45.3 0.48.0
ZEROs -> nightly bias no yes
DARK -> badcol mask no yes

Example failure run from an interactive batch node:

source $CFS/desi/software/desi_environment.sh test-21.12
srun -N 1 -n 21 -c 2 desi_proc --mpi --traceshift -n 20210507 -e 87539 --cameras b5
...
WARNING cholesky_solve failed with status 351
FATAL ERROR (other std) FitSeveralSpots failed for FLUX+TRACE (at line 2562 of file /tmp/pip-req-build-lryb6298/src/specex_psf_fitter.cc)
...
ERROR:specex.py:194:main: desi_psf_fit on process 18 failed with return value 1 running desi_psf_fit -a /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/preproc/20210507/00087539/preproc-b5-00087539.fits --in-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/exposures/20210507/00087539/shifted-input-psf-b5-00087539.fits --out-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/exposures/20210507/00087539/fit-psf-b5-00087539_18.fits --lamp-lines /global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/code/specex/0.8.3/lib/python3.8/site-packages/specex/data/specex_linelist_desi.txt --first-bundle 18 --last-bundle 18 --first-fiber 450 --last-fiber 474 --legendre-deg-wave 1 --broken-fibers 342
...

Note: desihub/desispec#1528 reports a bug about the desi_spec batch script generation when running specex on a single camera, thus the "by hand" command run from an interactive batch node above.

@julienguy @marcelo-alvarez please track this down to get these working again; this is a blocking factor for Fuji.

julienguy commented 2 years ago

I do not have the FLUX+TRACE after masking fiber 2900 which is on a dead column (I am committing the change to DESI_SPECTRO_CALIB). But for the specific example given, the readnoise on amp B is such that the code (rightfully) discards the PSF.

ERROR:proc.py:490:main: 25% of b5 input pixels have bad readnoise; don't use this PSF
julienguy commented 2 years ago

For those nights in May where b5 was very noisy, we can use the default PSF $DESI_SPECTRO_CALIB/spec/sm9/psf-sm9-b5-20210912.fits. The difference is typically of 1-2% with the daily PSF (from a comparison with the PSF $DESI_SPECTRO_REDUX/daily/calibnight/20210507/psfnight-b5-20210507.fits which was completed despite the high readnoise, and now masked out with the recent software). It is not optimal but is definitely better than discarding nights of data.

julienguy commented 2 years ago
julienguy commented 2 years ago

The read noise threshold issue is addressed in desispec issue https://github.com/desihub/desispec/issues/1559 so we may close this one (specex 59) when both desispec PR #1564 issue #1559 are merged/addressed.

sbailey commented 2 years ago

another contributing factor to the b5 failures is a huge number of incorrectly identified bad columns (see desihub/desispec#1567). I think all of the relevant pieces are being tracked in other tickets so I'm going to close this one.