pixel model and chi2 metrics for better cosmic identification

[sbailey]

This PR adds several new outputs to the ex2d() spectral extractions:

1. pixel model of the extracted spectra projected back into the CCD pixel space, useful for debugging
2. the PSF-weighted fraction of input pixels that had ivar=0, useful for identifying ill-constrained fluxes (perhaps to be masked by downstream steps)
3. the PSF-weighted chi^2 of the pixels that contributed to each extracted flux bin, useful for identifying bad extractions, e.g. arising from unmasked cosmics in the input pixel data

These are provided as informational numbers that downstream steps could use for masking. A future specter improvement might use the pixel-level chi2 to identify and mask poorly fitting pixels (e.g. from cosmics) and then re-extract, potentially "saving" some flux bins that otherwise would have been lost as just bad extractions. But that iterative refit isn't included in this PR.

The ipython notebook in doc/nb/unmasked_cosmics.ipynb documents more details about how this works. The following figure shows an unmasked cosmic in the DESI oak simulations that was incorrectly identified as H-alpha; the upper plot shows the extracted spectrum and the wavelengths that would be masked by applying a cut on the PSF-weighted pixel-chi2.

Testing on the DESI oak-lite simulations (2 spectrographs, 3 exposures) using this with a conservative 10-sigma cut improved the ELG redshifts from 90% efficiency to 94%, and LRGs from 97% to 99%. Full disclosure: compared to oak (10 spectrographs, 9 exposures) the QSO efficiency went down, but the efficiencies were already so low (50-60%) something else is going on that needs to be debugged independent of this.

Ted and Julien: I had previously reported that this was blowing memory on edison batch jobs. That turned out to be a leftover of caching the results of every subpatch extraction for debugging. Removing that, the memory footprint is only modestly larger (1.5 vs. 1.4 GB).

I'll submit a PR on the desispec side to use these features.

[julienguy]

• in ex2d.ex2d line 158
  • need to ignore masked (ivar=0) pixels in PSF weighted chi2 (otherwise flux bins affected with partially masked cosmic rays will have low weighted chi2).
  • need to test A is not zero for some values. I got a warning when running test_extract.py
• would need to allow for a psf relative error (few %), otherwise, there is a risk to flag as bad very bright spectral bins.

[julienguy]

code tested successfully (along with desispec branch cosmic_masking) on one cosmic ray spike (oak1 frame r0-00000006 fiber 413 wave 6519A). with new code, cosmic shows very large CHI2PIX value in frame, and is adequately flagged in MASK.

[tskisner]

I defer to Julien's detailed testing on this...

[sbailey]

These are good comments. I'm about to leave on vacation so I'm not going to try to fix them and merge in a rush and then disappear. Will wait for late August...

[sbailey]

I added changes to address Julien's comments:

• chi2pix calculation now includes a PSF model error. PSFs can specify their error via a PSFERR keyword in the fits file. If that isn't specified a default of 0.01 is used. This is a global PSF error; wavelength and fiber dependent PSF errors are not supported (yet?)
• chi2pix calculation doesn't include masked pixels
• avoid divide by 0 warning

I updated the ipython notebook and re-tested the example shown in the original PR post (which still works fine...)