search

desihub / specex

DESI spectrograph PSF fitting
BSD 3-Clause "New" or "Revised" License
0 stars 4 forks source link

FATAL ERROR ... cannot deal with varying bundle size #30

Closed sbailey closed 4 years ago

sbailey commented 4 years ago

When processing night 20191109 exposure 25728 via desi_proc, b1 fails due to

FATAL ERROR (harp) Exception at line 248 of file specex_trace.cc:  cannot deal with varying bundle size

desispec/master + spexec/master + desi_spectro_calib/trunk, via

srun -n 320 -c 2 desi_proc -n 20191109 -e 25728 --mpi

desi_proc itself is problematic that it gets hung on the error rather than cleaning catching it an moving on like the full pipeline would. i.e. when debugging this don't wait for desi_spec to finish.

julienguy commented 4 years ago

What is the input PSF file?

sbailey commented 4 years ago

Input PSF /global/project/projectdirs/desi/spectro/desi_spectro_calib/trunk/spec/sp1/psf-sm2-b-science-slit-20191107.fits . Curiously when I tried to reproduce it by running a single camera it failed but with a different error:

source /project/projectdirs/desi/software/desi_environment.sh master
module swap specex/master
srun -N 1 -n 20 -c 2 -C haswell -q interactive -t 00:20:00 desi_proc --mpi --cameras b1 -n 20191109 -e 25728
...
FATAL ERROR (harp) Exception at line 2884 of file specex_psf_fitter.cc:  FitSeveralSpots failed for FLUX+TRACE
ERROR:specex.py:218:main: desi_psf_fit on process 19 failed with return value 1 running desi_psf_fit -a /global/project/projectdirs/desi/spectro/redux/sjbailey/preproc/20191109/00025728/preproc-b1-00025728.fits --in-psf /global/project/projectdirs/desi/spectro/desi_spectro_calib/trunk/spec/sp1/psf-sm2-b-science-slit-20191107.fits --out-psf /global/project/projectdirs/desi/spectro/redux/sjbailey/exposures/20191109/00025728/psf-b1-00025728_19.fits --first-bundle 19 --last-bundle 19 --first-fiber 475 --last-fiber 499 --broken-fibers 349,473,474
...
julienguy commented 4 years ago

/global/project/projectdirs/desi/spectro/desi_spectro_calib/trunk/spec/sp1/psf-sm2-b-science-slit-20191107.fits has 20 bundles of 25 fibers, so your first run must have used the testslit PSF which has indeed uneven bundles and need to be run with a specific option ( --single).

sbailey commented 4 years ago

I verified that this works with the latest PSF files in $DESI_SPECTRO_CALIB, so closing this ticket.