search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

trace shifts ValueError: Zero or negative bin size ? #2246

Closed sbailey closed 3 months ago

sbailey commented 4 months ago

Jura job arc-20220220-00122956-a0123456789-25357199.log on node nid004324 starting at Thu 09 May 2024 03:10:40 PM PDT failed with

CRITICAL:util.py:155:runcmd: FAILED rank 0 exception while running desispec.scripts.trace_shifts.main(['-i', '/dvs_ro/cfs/cdirs/desi/spectro/redux/jura/preproc/20220220/00122956/preproc-b4-00122956.fits.gz', '--psf', '/dvs_ro/cfs/cdirs/desi/spectro/desi_spectro_calib/0.6.0/spec/sm1/psfmean-b4-20220101-20220130.fits', '--degxx', '0', '--degxy', '0', '--degyx', '0', '--degyy', '0', '--arc-lamps', '--outpsf', '/global/cfs/cdirs/desi/spectro/redux/jura/exposures/20220220/00122956/shifted-input-psf-b4-00122956.fits']) called from line 612 in /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/scripts/proc.py
ERROR:util.py:163:runcmd: Traceback (most recent call last):
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/util.py", line 146, in runcmd
    result = cmd(*args)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/scripts/trace_shifts.py", line 386, in main
    tset = fit_trace_shifts(image=image, args=args)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/scripts/trace_shifts.py", line 345, in fit_trace_shifts
    tset.y_vs_wave_traceset._coeff = shift_ycoef_using_external_spectrum(psf=psf,xytraceset=tset,
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/trace_shifts.py", line 734, in shift_ycoef_using_external_spectrum
    ref_spectrum = resample_flux(wave, ref_wave , ref_spectrum)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/interpolation.py", line 113, in resample_flux
    return _unweighted_resample(xout, x, flux, extrapolate=extrapolate)
ERROR:util.py:163:runcmd: File "/global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.63.0/lib/python3.10/site-packages/desispec/interpolation.py", line 225, in _unweighted_resample
    raise ValueError("Zero or negative bin size")
ERROR:util.py:163:runcmd: ValueError: Zero or negative bin size

However, when I re-run that command by hand (with a different output file location) it doesn't crash. I'm submitting this ticket to get the error code into our searchable history, then I'll re-submit the job.

sbailey commented 3 months ago

Resubmitted job also worked fine; closing ticket.

sbailey commented 1 month ago

This also happened for Kibo arcs

CRITICAL:util.py:155:runcmd: FAILED rank 0 exception while running desispec.scripts.trace_shifts.main(['-i', '/dvs_ro/cfs/cdirs/desi/spectro/redux/kibo/preproc/2021
1108/00107907/preproc-z1-00107907.fits.gz', '--psf', '/dvs_ro/cfs/cdirs/desi/spectro/desi_spectro_calib/0.7.0/spec/sm10/psf-sm10-z1-20210912.fits', '--degxx', '0', 
'--degxy', '0', '--degyx', '0', '--degyy', '0', '--arc-lamps', '--outpsf', '/global/cfs/cdirs/desi/spectro/redux/kibo/exposures/20211108/00107907/shifted-input-psf-
z1-00107907.fits']) called from line 612 in /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.0/lib/python3.10/site-packages/desis
pec/scripts/proc.py

and

CRITICAL:util.py:155:runcmd: FAILED rank 0 exception while running desispec.scripts.trace_shifts.main(['-i', '/dvs_ro/cfs/cdirs/desi/spectro/redux/kibo/preproc/2021
1111/00108365/preproc-z1-00108365.fits.gz', '--psf', '/dvs_ro/cfs/cdirs/desi/spectro/desi_spectro_calib/0.7.0/spec/sm10/psf-sm10-z1-20210912.fits', '--degxx', '0', 
'--degxy', '0', '--degyx', '0', '--degyy', '0', '--arc-lamps', '--outpsf', '/global/cfs/cdirs/desi/spectro/redux/kibo/exposures/20211111/00108365/shifted-input-psf-
z1-00108365.fits']) called from line 612 in /global/common/software/desi/perlmutter/desiconda/20240425-2.2.0/code/desispec/0.66.0/lib/python3.10/site-packages/desis
pec/scripts/proc.py

both worked fine when run in isolation, and the jobs also worked when resubmitted.