4.3% of sky fibers have ZWARN==0

[londumas]

Using the full performance of redrock on eBOSS, we have: 3,059,850 fibers, 282,341 (9.2%) are sky fibers, 24,100 have ZWARN==0 (8.5%, 0.8% of total) Redrock classifies these sky fibers as mostly galaxies. This is logical since the galaxy template has 10 PCAs, when stars have 5 and QSOs has 4.

CLASS  || all   || ZWARN==0 ||
STAR   || 0.4%  || 1.7%     ||
GALAXY || 97.8% || 95.1%    ||
QSO    || 1.8%  || 3.2%     ||

Here are the fake redshift distribution for these three classes. It is possible that some sky-fibers have objects in them but it wouldn't produce sharp distributions.

The list of eBOSS sky fibers is in: /global/homes/h/hdumasde/Run_programs/desi/redrock/look_sky_fibers/list_sky_fibers.txt

[moustakas]

In PRIMUS we included a power-law "template". I.e., if a spectrum was indistinguishable from a power-law then we flagged this with a dedicated zwarning flag. I wonder if many of these false-positives would be flagged by something along these lines.

[londumas]

@moustakas, The issue with this method is that the mis-calibration of the sky results in apparent emission lines or anti-emission lines. This method you suggest only uses broadbands. It removes some sky fibers but not all because the galaxy template has very sharp emission lines that can fit better the sky mis-calibrated lines. I had a look.

[londumas]

We could also play on the minimum Delta chi2 between two best fit. Currently it is set to 9.

[londumas]

Here is a small study of the completeness and purity vs. DeltaChi2Min for the three classes with full eBOSS. In each plot the blue curve gives the cumulative number of objects with ZWARN==0 or ZWARN==SMALL_DELTA_CHI2. The orange line gives the percent of these objects not being a sky fiber. The black line is the 99% purity. Currently DeltaChi2Min=9

If we target 99% purity, then:

• STAR: DeltaChi2 min ~ 4, only 90% objects survive
• GALAXY: Delta chi2 min ~ 50, only 78% objects survive
• QSO: Delta chi2 min ~30 , only 94% objects survive

If we target 95% purity, then:

• STAR: DeltaChi2 min = 0, all objects survive
• GALAXY: Delta chi2 min ~ 30, only 80% objects survive
• QSO: Delta chi2 min ~ 14, only 96% objects survive

[londumas]

Thanks to archetypes, this number dropped by more than half: from 10% to 4.3%. The current confusion matrix for ZWARN==0 sky fiber is:

                         | GALAXY  | QSO     | STAR    | 
SKY                14888 |  71.43  |  17.09  |  11.49  | 

Here are the distribution of sky fibers:

[moustakas]

This is a very nice study but not really an open ticket anymore. I've added a note to the DR1 paper where we could consider including this type of analysis.