Add support for reading plug map files

dkirkby commented 8 years ago

These files are available via SAS under apo/mapper/MJD/plPlugMapM-PLATE-MJD-VER.par where the version VER seems to be always 01, e.g.

http://dr12.sdss3.org/sas/dr12/apo/mapper/52904/plPlugMapM-1089-52904-01.par

dkirkby commented 8 years ago

The generic plug map data model is here. The file name template is plPlugMapM-YYYY-ZZZZZ-AAQ.par with:

YYYY = plate id
ZZZZZ = MJD of the mapper run
AA = "unique mapped-plugged-plate id" (always 01 ?)
Q = pointing identifier A,B,C,... or omitted for plates with only a single pointing (all BOSS plates?)

Questions:

Is the mapper MJD the same as the observation MJD? If not, it is stored in the plate metadata somewhere?
Can we safely assume that AAQ = "01" ?

dkirkby commented 8 years ago

Plate 6641 has an observation MJD of 56383. Check the mapper dir for this MJD:

http://dr12.sdss3.org/sas/dr12/apo/mapper/56383/

It does not contain plate 6641. Hunting backwards, I found 6641 was mapped 12 days earlier:

http://dr12.sdss3.org/sas/dr12/apo/mapper/56371/

Note that the plug map file for 6641 has AAQ = "02" with no "01" present in the directory, so we cannot always assume "01":

http://dr12.sdss3.org/sas/dr12/apo/mapper/56371/plPlugMapM-6641-56371-02.par

dkirkby commented 8 years ago

Plug maps are also available in the SVN data/sdss/platelist, e.g. for plate 6641:

https://trac.sdss.org/browser/data/sdss/platelist/trunk/plates/0066XX/006641

Note that this is a PlugMapP file, not a PlugMapM file. Here is a sample guide fiber line, for comparison between these files:

PlugMapP

PLUGMAPOBJ { 3818 301 6 146 68 } GUIDE 180.15847 45.862423
  { 16.9845 15.8033 15.3328 15.2381 15.2116 } 0.00000 0.00000 0.00000 NA
  -280.05378 -1.4064334 0 1 0 0 0

PlugMapM

PLUGMAPOBJ { 3818 301 6 146 68 } GUIDE 180.15847 45.862423
  { 16.9845 15.8033 15.3328 15.2381 15.2116 } 0.00000 0.00000 0.00000
  NA -280.05378 -1.4064334 0 1 0 0 0

These look identical.

dkirkby commented 8 years ago

The plug map filename PLATE-MJD-VER is stored as NAME in the HDU0 header of the raw data sdR files, so this is related to #29.

dkirkby commented 8 years ago

The pipeline uses version plPlugMapM files in the svn speclog product. For example, the plug map for 6641 is at:

https://trac.sdss.org/browser/data/sdss/speclog/trunk/56371/plPlugMapM-6641-56371-02.par

and has the same contents, e.g.

PLUGMAPOBJ { 3818 301 6 146 68 } GUIDE 180.15847 45.862423
  { 16.9845 15.8033 15.3328 15.2381 15.2116 } 0.00000 0.00000 0.00000
  NA -280.05378 -1.4064334 0 1 0 0 0

dkirkby commented 8 years ago

The guide fiber info could be easily scraped with regexps, but there is already a python package to parse the yanny format:

http://www.sdss3.org/svn/repo/yannytools/trunk/python/yanny.py

This is a single-file package whose only external dependency is numpy.

dkirkby commented 8 years ago

It looks like reading the speclog package will be the easiest way forward, even though it is outside of the SAS tree. Info about using the project SVN repo is here. Access generally requires a trac account, but some products (including speclog) are available through a public SVN portal. To check out a read-only copy of speclog, use:

svn co https://svn.sdss.org/public/data/sdss/speclog/trunk speclog

Should we try to automate this or require that users do this manually and then pass the speclog directory via an envvar or function args? Best to start with doing it manually.

dkirkby commented 8 years ago

Public svn access to speclog is currently broken:

% svn co https://svn.sdss.org/public/data/sdss/speclog/trunk/ speclog
svn: E175013: Unable to connect to a repository at URL 'https://svn.sdss.org/public/data/sdss/speclog/trunk'
svn: E175013: Access to 'https://svn.sdss.org/public/data/sdss/speclog/trunk' forbidden

dkirkby commented 8 years ago

In the meantime, checkout a non-public copy using:

svn checkout https://svn.sdss.org/data/sdss/speclog/trunk speclog --username dkirkby

Details are here.

The checkout takes ~15 mins to complete and creates a ~25 Gb directory.

dkirkby commented 8 years ago

The speclog checkout contains MJDs starting at 51078 (22-Sep-1998) and going up to the present, so it does not distinguish between SDSS-I ... IV.

dkirkby commented 8 years ago

Remove the ~11Gb of guidermon files using:

cd speclog
find . -name 'guiderMon-*.par' -delete

dkirkby / bossdata

Add support for reading plug map files #103