PSF writes file despite FATAL ERROR

sbailey commented 2 years ago

This command generates two "FATAL ERROR" messages and writes bundles 00 and 03 with 3x3 PSFs instead of 8x5 PSFs, before failing upon merging:

OUTDIR=/global/cfs/cdirs/desi/users/users/$USER/psftest
mkdir -p $OUTDIR
REDUXDIR=/global/cfs/cdirs/desi/spectro/redux/daily
export OMP_NUM_THREADS=3
srun -n 20 -c 3 desi_compute_psf_mpi \
   --input-image $REDUXDIR/preproc/20211005/00103082/preproc-z6-00103082.fits \
   --input-psf $REDUXDIR/exposures/20211005/00103082/shifted-input-psf-z6-00103082.fits \
   -o $OUTDIR/fit-psf-z6-00103082.fits &> $OUTDIR/psf-z6-00103082.log

in the logfile:

...
WARNING cholesky_solve failed with status 43
FATAL ERROR (other std) FitSeveralSpots failed for FLUX+TRACE (at line 2562 of file /tmp/pip-req-build-6u6z5d3_/src/specex_psf_fitter.cc)
...
WARNING problem with brent dchi2 = -1.38853e+07
FATAL ERROR (other std) FitSeveralSpots failed for FLUX+TRACE (at line 2562 of file /tmp/pip-req-build-6u6z5d3_/src/specex_psf_fitter.cc)
...
INFO:specex.py:364:merge_psf: Will merge 20 PSFs in /global/cfs/cdirs/desi/users/users/sjbailey/psftest/fit-psf-z6-00103082.fits
INFO:specex.py:369:merge_psf: merging /global/cfs/cdirs/desi/users/users/sjbailey/psftest/fit-psf-z6-00103082_01.fits into /global/cfs/cdirs/desi/users/users/sjbailey/psftest/fit-psf-z6-00103082_00.fits
srun: error: nid00015: task 0: Floating point exception
srun: launch/slurm: _step_signal: Terminating StepId=50602372.1
slurmstepd: error: *** STEP 50602372.1 ON nid00015 CANCELLED AT 2021-11-18T20:49:42 ***
srun: error: nid00015: tasks 1-19: Terminated
srun: Force Terminated StepId=50602372.1

Several problems / mysteries:

why is it failing in the first place (e.g. do we need to update a CCD mask?)
if there is a FATAL ERROR it should stop instead of writing a PSF
Two of the bundle PSFs it did write are 3x3 instead of 8x5; how did it get to the point of writing that incorrectly-sized PSF?
The log does not have "FAILED: MPI group ..." error messages that would have appeared if desispec.scripts.specex.main had raised an exception (see https://github.com/desihub/desispec/blob/master/py/desispec/scripts/specex.py#L320-L327)
When the nightly pipeline ran this command wrapped by desi_proc (log in /global/cfs/cdirs/desi/spectro/redux/daily/run/scripts/night/20211005/arc-20211005-00103082-a0123456789-50140129.log, there was one "FATAL ERROR" but not two, and the merging apparently "succeeded" albeit writing a bad 3x3 PSF to /global/cfs/cdirs/desi/spectro/redux/daily/exposures/20211005/00103082/fit-psf-before-listed-fix-z6-00103082.fits .
- I have not been able to reproduce that case with direct calls to desi_compute_psf_mpi on Haswell or KNL at various OMP_NUM_THREADS settings; I have not tried rerunning the full PSF from the beginning with the desi_proc wrapper.

@marcelo-alvarez

marcelo-alvarez commented 2 years ago

Please see #57 for a solution to this (except for why the fit is failing in the first place, which others may be in a better position to investigate than I am). It turns out that #54 did not actually fix this problem when it came up last time, because the test I did failed to include the catching of the "fatal error" exception (in fit_psf) the prevented it from being caught . See #53 for more details in the full context.

sbailey commented 2 years ago

fixed in #57 (still writes per-bundle output files, but provides the hooks to not do the merge to the final file)

desihub / specex

PSF writes file despite FATAL ERROR #56