Some spFrame files have no corresponding spCFrame files

[dkirkby]

For example, exposure 158840 of plate 6641 is the first one mentioned in the spPlancomb file, but does not have a corresponding spCFrame file. As a result, the second of these commands fails with an HTTP 404 error:

bin/bossplot --frame --plate 6641 --mjd 56383 --fiber 30 --exposure 0 --camera both --save frame.png
bin/bossplot --cframe --plate 6641 --mjd 56383 --fiber 30 --exposure 0 --camera both --save cframe.png

We currently use two sources to build a list of exposures for a plate:

1. The spFrame files listed in a spPlancomb file.
2. The header keywords in a spec file.

The first seems to be a complete list of the available spFrame files and the second lists all spCFrames actually used in the final coadd. What we are missing is a method to determine which spCFrame files should be present in the plate directory.

[dkirkby]

This issue was discovered while testing for #18

[dcunning11235]

Another example: plate 3606, exposure 105005

[dcunning11235]

After running into this again and not realizing it was, in fact, again, did some investigation.

Looking only in the plate directory e.g. http://data.sdss3.org/sas/dr12/sdss/spectro/redux/v5_7_0/3794/ and spectra directory http://data.sdss3.org/sas/dr12/sdss/spectro/redux/v5_7_0/spectra/3794/ there appear to be a number of possible sources for the CFrames actually available: 1.) The directory listing, filtered for "spCFrame-.fits" (Small, but not a file... unless we crawl this and roll out own.) 2.) The individual spec file headers (~2MB) 3.) The spPlate file headers (~110 MB) 4.) Getting the list of frame (below) and removing any rejected in spDiagcomb*.log (~13-14 MB)

The frames available: 1.) The directory listing, filtered for "spFrame-*.fits.gz" 2.) spPlan2d and spPlancomb files

There may be other sources of this information not in the plate or specta directories, but I am not aware of it. Given the above, then, it seems like the best option (smallest file, least hacky) is to download e.g. the fiber 0001 spec file and read the CFrames out of that.

[dkirkby]

It feels like a hack to download a speclite file just to get a list of exposures, but I don't have a better idea.

[dkirkby]

This is what I am using to iterate over all exposures actually used in a coadd:

    assert camera in ('b1', 'b2', 'r1', 'r2')
    band = 'blue' if camera in ('b1', 'b2') else 'red'
    spec_id = int(camera[1])

    # Use a speclite file to get the list of exposures actually used in the co-add.
    num_fibers = bossdata.plate.get_num_fibers(plate)
    first_fiber = (spec_id - 1) * (num_fibers // 2) + 1
    spec_name = finder.get_spec_path(plate, mjd, fiber=first_fiber, lite=True)
    spec_file = bossdata.spec.SpecFile(mirror.get(spec_name))

    # Loop over science exposures used in the co-add.
    for exposure_index in range(spec_file.num_exposures):
        # Open the calibrated frame file for this camera.
        cframe_name = finder.get_plate_path(
            plate, spec_file.get_exposure_name(exposure_index, band, 'spCFrame'))
        cframe = bossdata.plate.FrameFile(mirror.get(cframe_name))

Note that this uses some API changes that have not been merged to master yet, but will be in v0.2.7.