more robust to zqso failures

sbailey commented 3 years ago

This PR adds robustness to make_zqso_files. Previously if a call to create_zcat failed for a particular tile,night,petal it could cause the program to stop without processing the remaining tile,night,petals requested. Now it prints the traceback, logs an error, then proceeds with the others. The exit code is now the number of failures.

An example case for triggering this is tile 325 night 20210406 where the pipeline succeeded in making a zbest file but failed at making a coadd file for petal 2. I'll separately investigate what happened there, but the point is that shouldn't block make_zqso_files from processing petals 3-9 just because it was missing inputs for petal 2.


<cori zqso> make_zqso_files -t 325 -n 20210406 -o $CSCRATCH/desi/qzsotest
...
INFO:make_zqso_files:144:<module>: processing TILEID=325, NIGHTID=20210406, petal=2
INFO:lyazcat.py:66:tmark: 
    Making redrock zcat: 2021-06-04 | 13:10:25
INFO:lyazcat.py:100:make_new_zcat: Read /global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/325/20210406/zbest-2-325-thru20210406.fits
INFO:lyazcat.py:66:tmark: 
    Adding QuasarNP data: 2021-06-04 | 13:10:25
Traceback (most recent call last):
  File "/global/common/software/desi/users/sjbailey/desitarget/bin/make_zqso_files", line 147, in <module>
    create_zcat(input_dir, output_dir, tile=tile,
  File "/global/common/software/desi/users/sjbailey/desitarget/py/desitarget/lyazcat.py", line 775, in create_zcat
    zcat = add_qn_data(zcat, coaddfn, qnp_model, qnp_lines, qnp_lines_bal)
  File "/global/common/software/desi/users/sjbailey/desitarget/py/desitarget/lyazcat.py", line 244, in add_qn_data
    data, w = load_desi_coadd(coaddname)
  File "/global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/code/QuasarNP/0.1.0/lib/python3.8/site-packages/quasarnp/io.py", line 428, in load_desi_coadd
    with fitsio.FITS(filename) as h:
  File "/global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/conda/lib/python3.8/site-packages/fitsio/fitslib.py", line 477, in __init__
    self._FITS = _fitsio_wrap.FITS(filename, self.intmode, create)
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/325/20210406/coadd-2
-325-thru20210406.fits

ERROR:make_zqso_files:156:<module>: Tile 325 night 20210406 petal 2 failed; continuing
INFO:make_zqso_files:144:<module>: processing TILEID=325, NIGHTID=20210406, petal=3
INFO:lyazcat.py:66:tmark: 
    Making redrock zcat: 2021-06-04 | 13:10:25
INFO:lyazcat.py:100:make_new_zcat: Read /global/cfs/cdirs/desi/spectro/redux/daily/tiles/cumulative/325/20210406/zbest-3-325-thru20210406.fits
...

coveralls commented 3 years ago

Coverage remained the same at 58.765% when pulling 3fd1de7a770841e949e2d7e1cfbc82d2eba72c71 on zqso_redux into de2c83ab48fd1981a7310e7c1ed3c7ba732bbaad on master.

sbailey commented 3 years ago

I'm going to self-merge this so that I can get it into production for running make_zqso_files on sv3 + main data.

desihub / desitarget

more robust to zqso failures #752