Closed moustakas closed 1 year ago
OK, this is definitely a bug. What happens is these objects are never even read because Z_RR<1e-3
, so they never have a chance to get their redshifts "corrected". I'm working on a fix now.
Another issue is that I appear to be using incorrect redshifts for 3299 objects (see below, building on the code above). Drilling down into the first object, it appears that the logic I'm using (https://fastspecfit.readthedocs.io/en/latest/vacs.html#updated-qso-redshifts) may be incorrect / incomplete:
qn = Table(fitsio.read('/global/cfs/cdirs/desi/spectro/redux/iron/healpix/main/dark/361/36183/qso_qn-main-dark-36183.fits'))
qn[qn['TARGETID'] == 39627340543690901]['TARGETID', 'Z_NEW', 'Z_RR']
<Table length=1>
TARGETID Z_NEW Z_RR
int64 float64 float32
----------------- ----------------- ---------
39627340543690901 2.536549385403353 1.7805845
Full sample:
jj['HEALPIX'] = jj['HPXPIXEL']
out = join(tt, jj, keys=['TARGETID', 'SURVEY', 'PROGRAM', 'HEALPIX', 'DESI_TARGET'], table_names=['FAST', 'LSS'])
assert(len(out)+len(miss) == len(jj))
diff = out[out['Z_LSS'] != out['Z_FAST']]['TARGETID', 'SURVEY', 'PROGRAM', 'HEALPIX', 'Z_LSS', 'Z_FAST', 'DESI_TARGET']
diff
<Table length=3299>
<Table length=3299>
TARGETID SURVEY PROGRAM HEALPIX Z_LSS Z_FAST DESI_TARGET
int64 str4 str4 int64 float64 float64 int64
----------------- ------ ------- ------- ------------------ ------------------- -------------------
39627340543690901 main dark 36183 2.536549385403353 1.7805844892920242 655394
39627357115385691 main dark 36527 0.626857834582093 1.9446037297101326 655458
39627357140554446 main dark 36530 1.1274715946805622 1.6260670972765672 655394
39627357719365101 main dark 36212 2.370761483167071 1.6487141423026246 655394
39627362802867407 main dark 17416 2.6943721468996578 1.9043881509138207 655394
39627362853199454 main dark 36526 2.3463482953972963 3.2562304333552183 4611686018428043362
39627368582619242 main dark 17411 1.9271584236034853 1.8342251149308004 655394
39627369064958461 main dark 36276 2.1988898383021307 1.513625845008762 655458
39627374936982135 main dark 36293 2.573733947715017 1.812128927914253 655394
39627375037649512 main dark 22538 2.138624622149523 1.5021115150445838 655394
39627380087588624 main dark 16757 0.5935763860655094 1.866448017977218 655394
39627386483904772 main dark 36295 1.6466358174239513 1.6152247224385443 655394
39627392175576356 main dark 36279 1.9377994162447914 5.63741184509524 655394
39627397821109996 main dark 36640 2.2597093836114177 1.5630537386538588 655458
39627398114707701 main dark 36219 5.126483126386998 5.17799143745699 1179778
39627398215371922 main dark 22539 2.1836807758694303 1.5008758140050233 655458
39627403638607415 main dark 36641 0.7637694268293987 2.1411360775987105 655458
... ... ... ... ... ... ...
39636353285097367 main dark 22681 2.7229406323123695 1.9633068476745965 655394
39636437234093926 main dark 22737 0.6943514171425593 2.017052092038529 655394
39636478124361246 main dark 17385 2.2028826505364596 1.5506542021004148 655458
39636485284038684 main dark 22766 2.9477605139352923 3.011673020867196 655394
39636515575305340 main dark 22922 2.5643468362578252 0.9521238940731278 655394
39636521615101921 main dark 22922 1.9610394046977955 2.0388997366605253 655394
39636557761614845 main dark 23121 2.057927341422701 1.4696118000228529 655394
39636584995232869 main dark 31145 3.2922356038522884 3.3097406415653707 655394
39636620919441149 main dark 31410 0.7081089774723569 2.0155123524353673 655394
39636649352628424 main dark 21959 1.6367039453254502 2.335067645429564 655394
39636671964124237 main dark 17871 2.27499952781342 2.262812289029426 655394
39636755615320847 main dark 19514 1.9733396641479715 1.9499487475460613 655394
39636755640488415 main dark 19513 2.9232141318006715 2.086047172557037 655394
39636761525094755 main dark 19594 1.43901641851695 2.040846745977564 655394
39636943272673627 main dark 8491 2.2434052702293714 1.5843428896089298 655394
39636974255997521 main dark 20019 2.4333791155817437 1.7228507451146609 655394
39637018577209037 main dark 8589 1.7052459068768118 0.15112310050866345 655394
39637295594210896 main dark 939 2.1980977735385685 1.5239411169369623 655458
So it looks like none of these 3299 objects are QSO targets, which is why they are excluded. @stephjuneau can you comment on how I should be handling these objects?
from desitarget.targetmask import desi_mask
np.sum(desi_mask.QSO & diff['DESI_TARGET'] != 0)
0
Looking at the QSO maker code (?), it looks like ELG and BGS targets are also allowed to have their redshifts updated by QuasarNet: https://github.com/desihub/LSS/blob/main/py/LSS/qso_cat_utils.py#L351-L362
If this is the code that I should be using it would be great if it was written in a more modular way somewhere.
Here's that first object at the QuasarNet redshift. I guess it's not crazy that that's Lyman-alpha (note: 2-pixel smoothing on the left-hand side):
Hi @moustakas
Looking at the QSO maker code (?), it looks like ELG and BGS targets are also allowed to have their redshifts updated by QuasarNet: https://github.com/desihub/LSS/blob/main/py/LSS/qso_cat_utils.py#L351-L362
If this is the code that I should be using it would be great if it was written in a more modular way somewhere.
Yes, this is the correct code. By modular, do you mean having a separate function as part of QSO-maker or elsewhere in the desi software stack? I guess I'd suggest applying these rules first before cutting out on redshift to remove possible stellar contaminants.
A note on all of this, which might be helpful for context: We generally run on everything, so that we can catch QSO that were not targeted as QSO. The non-QSO target info is (so far, that I know of) only used by the GQP group (with lya showing some interest about anything high redshift). For the LSS catalogs, we only query the QSO info after we have already cut to only QSO targets. What is there for the other targets is, I think, still somewhat experimental (in terms of purity and redshift accuracy, at least last I knew, @stephjuneau should know better). Exactly what to do with it, I'm not sure. I don't believe we have an "official" answer for these within DESI at this point.
@moustakas could you please send me the full sample of ~3k above? I tried to copy/paste from GitHub but it didn't expand the truncated part of the table.
@stephjuneau here's the code (combined from a couple places above):
import numpy as np
import fitsio
from astropy.table import Table, join
tt = Table(fitsio.read('/global/cfs/cdirs/desi/spectro/fastspecfit/iron/catalogs/fastspec-iron-main-dark.fits', 'METADATA'))
jj = Table(fitsio.read('/global/cfs/cdirs/desi/survey/catalogs//Y1/QSO/iron/QSO_cat_iron_main_dark_healpix_v0.fits'))
jj['HEALPIX'] = jj['HPXPIXEL']
out = join(tt, jj, keys=['TARGETID', 'SURVEY', 'PROGRAM', 'HEALPIX', 'DESI_TARGET'], table_names=['FAST', 'LSS'])
diff = out[out['Z_LSS'] != out['Z_FAST']]['TARGETID', 'SURVEY', 'PROGRAM', 'HEALPIX', 'Z_LSS', 'Z_FAST', 'DESI_TARGET']
@moustakas I don't have a complete analysis yet but some first findings:
I'm attaching redshift distributions for the two values. The bottom panels zooms in around 0<z<1 and from this, there are 37 galaxies at low-redshift (z<0.2) with ZWARN=0. So looking at those 37 could be useful to identify whether there is a specific failure mode. BUT this is a tiny number given that this is for all of Iron-main-dark.....
Maybe a question for @dylanagreen : does QN have a lower redshift limit of z>0.5? Or would this be when RR is ran again with the QSO priors? It seems pretty abrupt on the redshift histograms above where Z_LSS comes from the LSS catalog for some QSOs identified with QN that were assigned a new redshift relative to RR.
Here's the first z=0.2 case with Z_FAST=0.199 but clearly the Z_LSS from QN is much better at z=1.8 with high confidence (there's even a MgII absorption doublet on top of the broad MgII, which I guess was confused for Ha+[NII] by RR). I'm on Team QSO for this one.
The second example with Z_FAST=0.16 actually now has Z_PIPE from the Daily reduction in better agreement with Z_LSS so again a QSO (although MgII is in an overlap region, both CIII] and CIV seem credible): https://www.legacysurvey.org/viewer-desi/desi-spectrum/daily/targetid39627575147891193
Here are more detailed notes on specific cases from my earlier VI of the first 25.
TARGETID SURVEY PROGRAM HEALPIX Z_LSS Z_FAST Z_DAILY COMMENT ON Z_DAILY
----------------- ------ ------- ------- ----------------- ------------------ ------- -------------------------------------------------
39627357719365101 main dark 36212 2.370761483167071 1.6487141423026246 2.3699 agrees with Z_LSS (QN)
39627362853199454 main dark 36526 2.346348295397296 3.2562304333552183 3.2204 disagress with both
39627374936982135 main dark 36293 2.573733947715017 1.812128927914253 1.0847 disagress with both
39627375037649512 main dark 22538 2.138624622149523 1.5021115150445838 2.1831 agrees with Z_LSS (QN)
39627397821109996 main dark 36640 2.259709383611418 1.563053738653859 0.0077 will be lost with DAILY (assigning to GALAXY now; Z_LSS is correct for this LAE)
39627398114707701 main dark 36219 5.126483126386998 5.17799143745699 5.0000 all are wrong; correct z=1.3076 ([OII] doublet)
39627398215371922 main dark 22539 2.183680775869430 1.50087581400502 2.1840 agrees with Z_LSS (QN)
39627403638607415 main dark 36641 0.763769426829399 2.1411360775987105 2.1431 agrees with Z_FAST (could be z=2.1337)
39636353285097367 main dark 22681 2.722940632312369 1.9633068476745965 0.2331 all are wrong; LAE at z=2.77 (now DAILY assigning to GALAXY)
39636478124361246 main dark 17385 2.202882650536460 1.5506542021004148 0.0020 Z_LSS is closer but both are wrong; DAILY says STAR but ZWARN=BAD_MINFIT; correct z=2.2455 (LAE)
@stephjuneau thank you for the detailed analysis.
I agree there are some improvements in the redshifts for a small number of non-QSO targets, but I would prefer to not bring in a dependency on the MgII after-burner fitting results. I personally think that all of these cases can be improved through better spectrophotometric templates.
Since I fixed the original bug (which was missing genuine QSOs and their updated QuasarNet redshifts), I would like to close this ticket. However, is there somewhere else we can move your VI work? Maybe https://github.com/desihub/desigal/issues?
@ashleyjross @paulmartini @geordie666 @stephjuneau @VFawcett
Before merging #148, I'd like to confirm that I'm using the correct QSO redshifts. My current procedure is documented here, which is supposed to mimic the choices made when generating the official QSO catalog-- https://fastspecfit.readthedocs.io/en/latest/vacs.html#updated-qso-redshifts https://github.com/desihub/fastspecfit/blob/main/py/fastspecfit/io.py#L843-L873
However, I'm concerned that I'm missing some subtleties.
E.g. there are 1151 objects in
iron
where I appear to be using the (incorrect) Redrock redshift,Z_RR
, butZ_RR<1e-3
, and so the object never appears in theFastSpecFit
VAC for Iron.