Add functionality to override MTL ledger entries

[geordie666]

This PR creates the concept of an "override ledger" that can be used to override entries in the standard MTL ledgers. The functionality includes:

• An add_to_override_ledgers script that can be used to create (or add to) a set of HEALPixel-split override ledgers.
  • The override ledgers are created using information from a user-supplied file.
  • That file needs to be readable as an astropy Table and include, at a minimum, RA, DEC, TARGETID and any columns needed to override MTL ledger entries.
  • It should also be possible to just change the override ledgers by-hand for bespoke applications.
• Aforce_mtl_overrides script that can instantly force information from the override ledgers into the MTL ledgers.
  • This is useful to prevent having to wait until the MTL loop is run again before targets are overrode.
  • This concept is still WIP. We may or may not actually use it.
• The glue to process overrides automatically as part of the MTL loop.
  • Some targets (e.g. high-z quasars) will need to override the results from the pipeline every time through the loop...
  • ...otherwise information from the pipeline could reassert dominance over the override information.

[geordie666]

To hopefully illustrate the end-to-end process, I've added some files in /global/cscratch1/sd/adamyers/testmtl that can be used to test the code and examine the results, as follows:

# ADM note that to run this, if you're mirroring my CSCRATCH directory structure you will also need to copy the file in
# /global/cscratch1/sd/adamyers/testops/outexample/tiles-specstatus.ecsv
# across to a similarly constructed directory in your CSCRATCH space.

# ADM preliminaries: copy the inexample data to an outexample directory so as not to overwrite the inexample data.
cp -R /global/cscratch1/sd/adamyers/testmtl/inexample /global/cscratch1/sd/adamyers/testmtl/outexample

# ADM A file of provided overrides (in this case Jinyi's VI'ed high-z quasars).
fil=/global/cscratch1/sd/adamyers/testmtl/QSO_reobs_jinyi.fits

# ADM This creates a set of override ledgers (in /global/cscratch1/sd/adamyers/testmtl/outexample/main/secondary/dark/override) from Jinyi's file.
# Here, we wish to override the "Z" column from the spectro pipeline with "VI_z" from Jinyi's file...
# ... and have also bizarrely chosen to reset all priorities to 1918
add_to_override_ledgers $fil dark --colsub '{"Z":"VI_z"}' --mtldir /global/cscratch1/sd/adamyers/testmtl/outexample --secondary --valsub '{"PRIORITY":1918}'

# ADM The previous script tells us which pixels we touched. 
# We can now force the override information into the corresponding MTL ledgers.
# This is kept separate from the add_to_override_ledgers script to give us a chance to 
# examine the override ledgers carefully, and then carefully again, before forcing changes into the MTL ledgers.
force_mtl_overrides /global/cscratch1/sd/adamyers/testmtl/outexample/main/secondary/dark 6472,2298,11251,2465,2343,2307,8089,3812,3336

# ADM Later on, we complete observations on tile 2204 and run the automated MTL loop.
run_mtl_loop DARK --mtldir /global/cscratch1/sd/adamyers/testmtl/outexample --zcatdir /global/cfs/cdirs/desi/spectro/redux/daily

# ADM to check the history of what happened to an illustrative override TARGET on tile 2204:
grep 39632981408614565 /global/cscratch1/sd/adamyers/testmtl/outexample/main/secondary/dark/mtl-dark-hp-2298.ecsv

[geordie666]

@ashleyjross: It would be great if you could review the results in /global/cscratch1/sd/adamyers/testmtl/outexample and see if you can understand what's going on. Feel free to reach out to me over email or on Slack for more detailed discussion.

[geordie666]

Documenting an offline email thread: Ashley is comfortable with the look of this set-up for LSS studies. I'll likely merge this PR tomorrow morning.