Add secondaries to MTL ledgers without merging with primaries

[geordie666]

This PR sets up the infrastructure for (occasionally) adding new secondary targets without merging with primary targets. A guiding principle is that the new secondaries will always have a different TARGETID to prevent their behavior ever affecting a primary (and vice-versa). The PR includes:

• New targeting bits for the RR_LYRAE and MWS_FAINT_* secondary targets.
• Updates to secondary.select_secondary() to completely ignore primary targets, if requested.
  • Every target is then effectively treated as an OVERRIDE target.
  • A new value of TARGETID is created based on writing secondaries to a main2 directory.
  • This TARGETID can never duplicate a previously used primary or secondary TARGETID.
• A top-line add_secondary_no_merge script to run secondary.select_secondary() without considering primaries.
• Functionality to append to ledgers instead of writing new ones in io.write_mtl().
  • With an associated append option in the make_initial_ledger script.

Also, along for the ride, I fixed a minor bug when reading headers of .ecsv files with the function io.read_ecsv_header(). That function was reading information about columns prior to the official header information, now it strictly extracts only dictionaries after the meta keyword.

[geordie666]

Basic usage, as an aide-memoire:

add_secondary_no_merge $CSCRATCH/blat --scnddir $CSCRATCH/ADMsecondary (create new main2 file of secondaries)
make_initial_mtl_ledger $CSCRATCH/blat/dr9/1.2.2.dev5180/targets/main2/secondary/dark/main2targets-dark-secondary.fits --dest $CSCRATCH/blat/mtl --obscon DARK --numproc 1 --append (add to the dark secondary ledger)
make_initial_mtl_ledger $CSCRATCH/blat/dr9/1.2.2.dev5180/targets/main2/secondary/bright/main2targets-bright-secondary.fits --dest $CSCRATCH/blat/mtl --obscon BRIGHT --numproc 1 --append (add to the bright secondary ledger)

[geordie666]

The outputs from using this approach with the PV_DARK* targets and the new MWS_RR_LYRAE, MWS_FAINT_RED and MWS_FAINT_BLUE targets are in:

/global/cscratch1/sd/adamyers/blat/dr9/1.2.2.dev5180/targets/main2/secondary/

for the output target files and:

/global/cscratch1/sd/adamyers/blat/mtl/main/secondary/

for the updated/appended to ledgers.

I'm setting this to a WIP, both to give people time to vet these files and as I just realized that I'll now need to update the MTL loop to look for the main2 directory when matching back to secondary targets.

[coveralls]

Coverage decreased (-0.2%) to 57.776% when pulling a29220c3d5e6ecedfebcf05f7ff5b1d7a3b13e8b on ADM-sec-no-merge into 02f08a97f34471ae8497b5ffaf721579916b7cf7 on master.

[geordie666]

Minor update: After further consideration, I realized that I don't actually need to update the MTL loop to look for the main2 directory in desitarget, but fiberassign will need to match back to the main2 directory. So, I'm basically done with this PR, but I'm leaving it as a WIP while @araichoor tests the results of this PR for fiberassign.

To help with fba testing, I've mocked up a version of the file structure for added secondary targets (and actually added the added secondary targets!). The relevant MTL directory is:

/global/cscratch1/sd/adamyers/test-new-mtl-adds/mtl (this would be instead of where fba would typically look at NERSC, in: /global/cfs/cdirs/desi/survey/ops/surveyops/trunk/mtl; obviously I didn't want to mess with the official directory).

and I've added a link farm that looks like /global/cfs/cdirs/desi/target/catalogs/dr9/1.1.1/targets in /global/cscratch1/sd/adamyers/desi/target/catalogs/dr9/1.1.1/targets (with the important files being: /global/cscratch1/sd/adamyers/desi/target/catalogs/dr9/1.1.1/targets/main2/secondary/dark/main2targets-dark-secondary.fits and /global/cscratch1/sd/adamyers/desi/target/catalogs/dr9/1.1.1/targets/main2/secondary/bright/main2targets-bright-secondary.fits)