bossplot 4020-55332-425

[kgage]

The following command (with verbose output)

$ bossplot --plate 4020 --mjd 55332 --fiber 425 --verbose
Plotting combined spectrum for 4020-55332-0425.
Exposure summary:
 exp    arc    flat   b1  b2  r1  r2
------ ------ ------ --- --- --- ---
115230 115233 115232   1   0   8   0
115231 115233 115232   2   0   9   0
115353 115357 115356   3   0  10   0
115354 115357 115356   4   0  11   0
115355 115357 115356   5   0  12   0
115426 115429 115428   6   0  13   0
115427 115429 115428   7   0  14   0
Showing coadd of 14 blue+red exposures.
Coadd (AND) pixel mask summary:
 4074 LOWFLAT
   11 NEARBADPIXEL
  287 BRIGHTSKY
 4596 BADFLAT
    1 BADSKYCHI

Gives the following image:

Using

$ bossplot --plate 4020 --mjd 55332 --fiber 425 --wlen-range [4000:10000] --flux-range [-2:8]

gives this:

These are very different from the spectrum shown on the SDSS object explorer: at http://skyserver.sdss.org/dr12/en/get/SpecById.ashx?id=4526234538074632192

[dcunning11235]

@kgage If you dump the fits file with something quick'n'dirty like:

import astropy.table
file_loc = 'spec-4020-55332-425.fits'
table = astropy.table.Table.read(file_loc, hdu=1)
print(table)
print(table['and_mask'][table['and_mask'] == 0])

you'll see that most of the pixels are anded out (meaning the same pixel was judged bad in every exposure for those same reasons, encoded in bits) leaving only 14 or 15 wavelength buckets (if I counted right). This is, in brief, what the classes bossplot uses are seeing.

For instance 131076 = 10 0000 0000 0000 0100 looks common in there, so bits 2 and 17 are set, so by http://dr12.sdss3.org/datamodel/files/BOSS_SPECTRO_REDUX/RUN2D/PLATE4/spPlate.html BADFLAT and LOWFLAT.

I don't know what those mean. EDIT: This is somewhat helpful: http://www.sdss.org/dr12/algorithms/fluxcal/

In any case, skyserver is clearly not filtering these and, apparently, no zwarning flag is set for this (would conceivably be ZWARNING=2, from http://www.sdss3.org/svn/repo/idlutils/trunk/data/sdss/sdssMaskbits.par)

[dkirkby]

The verbose bossplot output above already gives a summary of the mask bits set for each pixel:

Coadd (AND) pixel mask summary:
 4074 LOWFLAT
   11 NEARBADPIXEL
  287 BRIGHTSKY
 4596 BADFLAT
    1 BADSKYCHI

You can also display masked regions on the plot if you add the --show-mask option. In this case, you also need to set the wavelength range by hand, to include the masked regions:

bossplot --plate 4020 --mjd 55332 --fiber 425 --verbose --show-mask --wlen-range [3500:10400]

[dmargala]

I think the issue here is if bossplot should show masked flux values? or at least have an option to override the default behavior.

On Tue, Jul 14, 2015 at 11:02 AM dkirkby notifications@github.com wrote:

The verbose bossplot output above already gives a summary of the mask bits set for each pixel:

Coadd (AND) pixel mask summary: 4074 LOWFLAT 11 NEARBADPIXEL 287 BRIGHTSKY 4596 BADFLAT 1 BADSKYCHI

You can also display masked regions on the plot if you add the --show-mask option. In this case, you also need to set the wavelength range by hand, to include the masked regions:

bossplot --plate 4020 --mjd 55332 --fiber 425 --verbose --show-mask --wlen-range [3500:10400]

[image: masked] https://cloud.githubusercontent.com/assets/185007/8680672/cbacde92-2a5a-11e5-9eee-b4f28e8dfafd.png

— Reply to this email directly or view it on GitHub https://github.com/dkirkby/bossdata/issues/83#issuecomment-121324619.

[dkirkby]

There are already two options to override the default behavior: --allow-mask and --show-invalid. The first lets you ignore specific mask bits, e.g.

bossplot --plate 4020 --mjd 55332 --fiber 425 --allow-mask 'LOWFLAT|BADFLAT'

The second option lets you see everything (included any unmasked ivar=0 pixels, if there are any):

bossplot --plate 4020 --mjd 55332 --fiber 425 --show-invalid

Note that the --show-invalid option implies --scatter since we cannot guarantee to have non-zero ivar for plotting flux errors.

[dmargala]

Those both seem very convenient! Should showing the masked values be the default behavior? Perhaps the word mask is being misappropriated (the spectrum looks reasonable to me in this case at least).

Side note: the noise appears maybe slightly larger compared to the SDSS object explorer plot.

There are already two options to override the default behavior: --allow-mask and --show-invalid. The first lets you ignore specific mask bits, e.g.

bossplot --plate 4020 --mjd 55332 --fiber 425 --allow-mask 'LOWFLAT|BADFLAT'

[image: allow] https://cloud.githubusercontent.com/assets/185007/8694444/fb79ce74-2ad6-11e5-9e1b-ebd9667a8a2e.png

The second option lets you see everything (included any unmasked ivar=0 pixels, if there are any):

bossplot --plate 4020 --mjd 55332 --fiber 425 --show-invalid

[image: invalid] https://cloud.githubusercontent.com/assets/185007/8694470/39e62acc-2ad7-11e5-8e4c-7cbad39f070c.png

Note that the --show-invalid option implies --scatter since we cannot guarantee to have non-zero ivar for plotting flux errors.

— Reply to this email directly or view it on GitHub https://github.com/dkirkby/bossdata/issues/83#issuecomment-121534275.

[dkirkby]

I agree that it is not clear why the flat was considered bad enough to mask, given what the unmasked data look like, but that's a problem with the pipeline, not this package.

I filed SDSS4 ticket #2407 about the fact that skyserver (and also, it turns out, the SAS webapp) appear to ignore the mask, but now I think our work here is done.