Update QSO templates for Y3

sbailey commented 5 months ago

Possible improvements could come from

DESI data for training set
NMF instead of PCA
Better training set coverage in color space
Upweighting rare QSOs in training set
Legendre nuisance terms
...

abrodze commented 4 months ago

rrtemplate-qso-HIZ-v1.1.fits was used with redshift prior on spectra classified as QSO to build a post facto redshift catalog addressing the lyman transmission bug (#237) in iron (see desihub/desida#11). v1.1 of HIZ quasar templates significantly improves redshift accuracy at z>2 in this configuration.

I ran Redrock (all templates classes, all redshifts) on 9351 DARK time objects and 5944 BRIGHT time objects from the iron redux to test v1.1 as a full replacement for v1.0.

DARK:

35 objects change SPECTYPE (no apparent trend)
For ziron > 2 (where IGM correction is applied): dz > 0.0033 = 6.42%; dz > 0.01 = 1.95%
For ziron > 1.4 (limit of HIZ templates): dz > 0.0033 = 4.51%; dz > 0.01 = 1.601%
for ziron < 1.4 (should be unaffected by change): dz > 0.0033 = 0.21%; dz > 0.01= 0.21%
50% of all redshift changes dz>0.01 have ZWARN!=0

BRIGHT:

3 objects change SPECTYPE (no apparent trend)
only 4 objects with dz > 0.0033 of which 2 have ZWARN!=0, all severe dz values

Redshift Comparision: ziron_vs_znew

Median and 90% CI of redshift changes for DARK time objects in velocity units. The gray vertical line is the lower limit of the HIZ QSO templates, and the black vertical line is where the Lyman correction begins. The direction and magnitude of changes are as expected -- z>2 redshifts should increase with v1.1 (labeled as new). dv_irontonew

sbailey commented 4 months ago

Thanks @abrodze . I think you told me this verbally, but for the written record: When you use rrtemplate-qso-HIZ-v1.1.fits to re-run just the HIZ QSOs in the same manner as you did for the Y1 post-processed QSO catalogs, you got identical results, correct?

for ziron < 1.4 (should be unaffected by change): dz > 0.0033 = 0.21%; dz > 0.01= 0.21%

Originally I thought this was a failing null-test, but from the plot it looks like all cases are LOZ templates getting superseded by the new HIZ template at a much different redshift (thus the two small dz cuts having the same percentage). We should expect some random swaps like this. Do you agree with that?

We'll still press forward with other QSO template methods, but this is a good check that we have the pieces prepared to use this as a backup option if things don't converge in time.

abrodze commented 4 months ago

Correct, rrtemplate-qso-HIZ-v1.1.fits can identically reproduce the zlya quasar catalog

I do not consider the dz>0.01 != 0 for z<1.4 a failure:

for 11 out of 16 of these spectra, the new redshift is correct with HIZv1.1 (assessed by me). A small piece of evidence that the optical depth correction has a positive impact beyond redshift refinement at z>2
all but 2 objects are true QSO with a trend of BAL features, weak emission, and/or red continuum. I would be optimistic about the QSO afterburners catching these if missed.
and I agree with what you said that the <1/3% rate for these swaps is expected from altering the chi2 surface with a new template

moustakas commented 4 months ago

I'm not sure this is the right place to post this ticket, but as part of the Y3 QSO template validation, we should check QSO (targets) which changed redshifts dramatically between EDR and Y1.

Here are a bunch of examples (all from sv3/dark) where the Iron redshift (Z_1) differs significantly from the EDR/QSO VAC (https://data.desi.lbl.gov/doc/releases/edr/vac/qso/, Z_2). In the case of, e.g., 39633404945238050 the z=2.53 EDR redshift is clearly correct, e.g.--- https://fastspecfit.desi.lbl.gov/target/sv3-dark-11227-39633404945238050

<Table length=113>
     TARGETID     HEALPIX        Z_1                Z_2
      int64        int32       float64            float64
----------------- ------- ------------------ ------------------
39627734057487088   25926  1.009201650305473 0.6257731858475949
39627739992425349   25916 0.9424157098667513 1.8529867771855788
39627746061585444   25911 2.3385580450836327  1.698593514619386
39627746082557098   25952  1.664497741782426 0.6656186354097829
39627751493208010   25598 0.8540720126029397   2.08098297915377
39627752067831529   25916 2.3494039934095774 1.2805737186607218
39627757528813172   25598 1.6493459562618404 1.0758672139830303
39627757553976712   25597 1.1176435719155111  1.459474746581493
39627758153765972   25954  2.520975179173588 1.3994634880841805
39627763644106977   26273 1.6262355167506828 1.9920639182787312
39627764109677389   26001 1.4988993797008148 1.7971159628091669
39627769688103314   26273 0.8474630128805303  1.986529641743608
39627770262717467   25956 0.8603602205748764 1.1134408569568268
39627770283693311   25935 1.3665703109879042  1.105267002383262
39627776218629011   26004 0.7879047793242844 1.1178755062710255
39627776294127236   25958 1.6279631744386225 0.8915384725663915
39627776306711613   25956 1.0146661699345079 1.6265238238558373
39627781771890241   26276 1.5746060317103265  2.223079597813426
39627788340167484   25961   0.72074531592403  1.323826574271902
39627793805349536   26283 1.0931948279616448 1.4445576877364994
39627793830514094   26284 1.4991464994440444 1.8814566844777036
              ...     ...                ...                ...
39633297344563215   11425 1.3848423327961614 0.8280293167015234
39633297361340067   11428 0.9958440440037698 1.9506730368510108
39633304562959222   11428 1.2448874563405998  2.662758553341163
39633311982685723   10150  1.028242123756326 1.3981774247117835
39633332794822301   11521 1.1649701729066246 1.6517283718708384
39633390156121756   11237 0.9856760801388535 1.8511546978693691
39633393150855689   11237 0.9378455940826855 1.4349456211417058
39633399744299131   10199 1.5506656884658132 1.1799902748570485
39633402009223714   11239 1.4982381576021147  1.961415514919373
39633404945238050   11227  1.777459621961331 2.5330306188629397
39633405561802504   10204   2.04555848447703 0.7052019357110813
39633407830917875   11227 2.1900073539105316 3.0160369111419785
39633419746935526   10207 1.6191296992699273 1.8117334069603896
39633422548731112   15352  2.170247402363762 0.7791359273792566
39633424666855785   11914 0.7529249234663337 1.8617091020091034
39633433336483389   10229 0.8931611238199374  2.334930998995938
39633433344871044   15354  1.410009065905783  1.104539139632127
39633456329657273   16040 1.0224777402715215 2.1423940018552874
39633458707825929   11606  0.758062926934109 1.4525966187791262
39633465888474460   16043  1.287199825550538 1.6738834801110731
39633465892668563   16043 1.6318182064384672 1.7555412649193844

stephjuneau commented 3 months ago

Hi @moustakas that specific example you posted indeed has a correct redshift in EDR (39633404945238050) but I would say the spectrum is more similar to typical LAE spectra than to the average QSO. Maybe there is a faint BL in CIV but Lya doesn't seem very broad; so maybe a less representative case for the QSO templates? This could be a fun list to VI though! EDIT: those are pretty faint so some of them might end up having a low VI confidence. To me the first one looks wrong in both, second one correct in EDR, third one more correct in DR1 but slightly off, etc. so it'll probably be a mixed bag.

desihub / redrock

Update QSO templates for Y3 #280