search

desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
33 stars 24 forks source link

qproc not using fallback option for calibrations in 2-ch readout #2203

Closed sybenzvi closed 3 months ago

sybenzvi commented 3 months ago

In tests of 2-ch CCD mode starting on 20240324, qproc raised exceptions for r4, r6, r7, and z7 due to calibration darks not being found. However, @julienguy created new darks in DESI_SPECTRO_CALIB and qproc should have found and used those since it has the --fallback-on-dark-not-found option enabled.

The error was "fixed" at KPNO using the following steps:

Processing is proceeding normally at KPNO but exceptions are still being raised at NERSC for the 2-ch mode.

The option --fallback-on-dark-not-foundshould cause qproc to transparently find the right calibration files in DESI_SPECTRO_CALIB and not raise exceptions even if calibrations are not present in DESI_SPECTRO_DARK. The cause is under investigation.

Waelthus commented 3 months ago

one (maybe obvious) thing to note here: when we use that fallback option, all DARK/BIAS frames will be taken from the DESI_SPECTRO_CALIB defaults, instead of using more recent darks. On NERSC things seem to run nominally again, picking up the DESI_SPECTRO_DARK calibs. If you point me to where things didn't fall-back when a working DESI_SPECTRO_CALIB was there, but not the correct DESI_SPECTRO_DARK dark/bias models I'll happily look at why the fallback fails...

sybenzvi commented 3 months ago

Thanks for the quick feedback @Waelthus. I find that when I process 2-ch exposures from 20240324 at NERSC, I need to unset the environment variable DESI_SPECTRO_DARK to eliminate qproc errors for r4, r6, r7, and z7. The data from last night (20240327) is processing normally.

I reproduced the issue at NERSC this morning using exposures from a couple of days ago:

It's odd because qproc finds appropriate calibration files when DESI_SPECTRO_DARK is turned off. If you read the logs, you see the fallback option is enabled.

sybenzvi commented 3 months ago

Noting here that I have updated DESI_SPECTRO_DARK at KPNO and re-enabled the environment variable for qproc.

I confirm that when DESI_SPECTRO_DARK is defined, qproc does not find the appropriate calibration data in DESI_SPECTRO_CALIB. It only worked at KPNO once I rsynced DESI_SPECTRO_DARK from NERSC.

Waelthus commented 3 months ago

ok, I think I found a potential cause, namely when raw.py inits a new calibfinder after the actual preprocessing happened. Can you test the PR?

sybenzvi commented 3 months ago

ok, I think I found a potential cause, namely when raw.py inits a new calibfinder after the actual preprocessing happened. Can you test the PR?

Yes, trying now.

sybenzvi commented 3 months ago

Still erroring out. The fallback option is being activated:

ERROR:calibfinder.py:563:find_darks_in_desi_spectro_dark: Didn't find matching r4 calibration darks in $DESI_SPECTRO_DARK, falling back to $DESI_SPECTRO_CALIB
WARNING: The unit 'nmgy' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
WARNING: The unit '1 / nmgy2' could not be saved in native FITS format and cannot be recovered in reading. It can roundtrip within astropy by using QTable both to write and read back, though one has to enable the unit before reading. [astropy.io.fits.convenience]
WARNING:fibermap.py:1270:annotate_fibermap: Overriding units for column 'PMRA': 'mas yr-1' -> 'mas yr^-1'.
WARNING:fibermap.py:1270:annotate_fibermap: Overriding units for column 'PMDEC': 'mas yr-1' -> 'mas yr^-1'.
INFO:image.py:84:write_image: iotime 1.177 sec to write preproc-r4-00232247.fits at 2024-03-28T10:17:10.380741
CRITICAL:calibfinder.py:559:find_darks_in_desi_spectro_dark: Didn't find matching r4 calibration darks in $DESI_SPECTRO_DARK, quitting
Traceback (most recent call last):
  File "/global/homes/s/sybenzvi/desi/desispec/bin/desi_qproc", line 10, in <module>
    sys.exit(qproc.main(qproc.parse()))
  File "/global/homes/s/sybenzvi/desi/desispec/py/desispec/scripts/qproc.py", line 158, in main
    cfinder = CalibFinder([image.meta,primary_header])
  File "/global/homes/s/sybenzvi/desi/desispec/py/desispec/calibfinder.py", line 360, in __init__
    self.find_darks_in_desi_spectro_dark(header)
  File "/global/homes/s/sybenzvi/desi/desispec/py/desispec/calibfinder.py", line 560, in find_darks_in_desi_spectro_dark
    raise IOError(f"Didn't find matching {camera} calibration darks in $DESI_SPECTRO_DARK, quitting")
OSError: Didn't find matching r4 calibration darks in $DESI_SPECTRO_DARK, quitting
Starting at Thu Mar 28 10:16:50 2024
RUNNING desi_qproc -i /global/cfs/cdirs/desi/spectro/data/20240324/00232247/desi-00232247.fits.fz --fibermap /global/cfs/cdirs/desi/users/sybenzvi/nightwatch/20240324/00232247/fibermap-00232247.fits --auto --auto-output-dir /global/cfs/cdirs/desi/users/sybenzvi/nightwatch/20240324/00232247 --cam r4 --fallback-on-dark-not-found
Done at Thu Mar 28 10:17:10 2024 (20.384597 sec)
Waelthus commented 3 months ago

ah, but that error is different. Added the option to inits in qproc as well in the PR, but if there's more things appearing it might be easier to reverse the meaning and make only the offline pipeline call an option to let things fail... Anyway, you can test again, but I'll not be available until tomorrow morning CET.

sybenzvi commented 3 months ago

@Waelthus , I had to fix a trivial typo but I think you've got it working. Running

desi_qproc -i /global/cfs/cdirs/desi/spectro/data/20240324/00232245/desi-00232245.fits.fz --fibermap /global/cfs/cdirs/desi/spectro/nightwatch/nersc/20240324/00232245/fibermap-00232246.fits --auto --cam r4

fails as expected, while

desi_qproc -i /global/cfs/cdirs/desi/spectro/data/20240324/00232245/desi-00232245.fits.fz --fibermap /global/cfs/cdirs/desi/spectro/nightwatch/nersc/20240324/00232245/fibermap-00232246.fits --auto --cam r4 --fallback-on-dark-not-found

correctly reverts to $DESI_SPECTRO_CALIB.