new QSO templates without LyA

[sbailey]

PR #104 corrects for mean LyA while doing the QSO redshift scan, but it still uses the old QSO templates from BOSS that didn't remove LyA while building them. Create a new set of QSO templates that also remove mean LyA while building the templates.

PR #104 showed an example of this, but there was a strange notch in the templates at LyA, appearing as if some of the input QSOs had the wrong z and were thus mis-corrected.

These updated templates along with #104 should provide more consistent results on real data, but should also be coordinated with an update to simqso (or the desispec wrapper of simqso) since those templates also include mean LyA, and then additional LyA absorption is included on top of that (ugh).

[londumas]

Here are the elements to produce new QSO PCAs from SDSS data:

• read newest SDSS data, maybe add low z Lya forest from HST data and high z Lya forest from XQ-100 = VLT/XSHOOTER legacy survey: https://arxiv.org/pdf/1607.08776.pdf, and look into APOGEE observations of low z quasars. Is it usefull to add other spactra from other programs? List is there: https://heasarc.gsfc.nasa.gov/w3browse/all/milliquas.html
• use DR12Q, where it agrees with QNET, use Z_PCA
• use a observed wavelength range ([3600,7235]?, how to choose which?)
• mask AND_MASK!=0
• mask sky lines (how to choose which?)
• cap the weight from IVAR in order to not be dominated by high SNR data. maybe simply cap the same way it is done in Lyman-alpha studies.
• correct the IVAR for the weight calibration found in Lya studies
• remove spectra with less than ~100 pixels
• correct the flux for the flux calibration found in Lya studies
• correct for the Galactic extinction
• correct for absorption of Lya, Lyb, Ly-gamma, ...
• re-scale spectra to median flux, remove spectra if median is ~<0
• use as many quasar as possible. What are the limitations? Memory? Running time?
• optimize the number of PCA

To get a better redshift, we can use a ladder from galaxies:

• start with low redshift quasars, where the second best redshift is a galaxy, and let's use the redshift of the galaxy from a simple custom fit to few high SNR sharp lines.
• compute PCA. It will not go as low as lambda_RF<900
• recompute PCA, with new redshift obtained with the new PCA basis
• use these PCA to get redshift of higher redshift quasar
• compute PCA. It will go to lambda_RF lower than the previous
• use these PCA to get redshift of even higher redshift quasar

Added three files to calibrate the ivar, the flux and veto some sky lines.

veto-lines.txt ivar-calib.fits.gz flux-calib.fits.gz

[sbailey]

I like these steps, with one suggestion to consider for the first step of building the PCA from an expanding redshift ladder: Consider tying the initial low redshift QSO sample to a custom fit of the sharp [OIII] 4959 and 5007 lines (and maybe Hbeta 4861) instead of using a second-best galaxy template fit redshift.

[moustakas]

It would be great to get these templates into desisim, not just redrock....

[moustakas]

@abrodze @dylanagreen can you please assess this ticket and close it if it's no longer relevant wrt the current set of QSO templates?

[dylanagreen]

@moustakas I believe, and @abrodze can correct me if I'm wrong, that her newly generated templates (the lya ones) correct for this effect up to using the Kamble/Calura model, and my templates that I've been working on use the Inoue model.

Based on this then any template set that will be used for Y3 (either the LYA ones or my NMF ones) will correct for this effect using one of the two aforementioned models, but that the templates used in Iron/Y1 did not correct for this effect, again hoping Allyson can correct me if this is wrong.

Probably since this issue is thus fixed moving forward (Y3 and beyond) I am in favor of closing it but I'll leave the final decision on this to someone else.