This PR begins setting up MTL to interface with tiles that have been reprocessed by the spectroscopic pipeline. This only partially completes the required work. As it will be easiest to test the code on real-world data, the DESI Operations team signed off on merging this partial work now while I further debug the logic. This will allow us to both recommence running the "standard" MTL loop and to begin adding information for reprocessed tiles to the tiles-specstatus.ecsv file after they have gone through QA.
This PR already achieves:
Finding tiles that have been reprocessed by checking for where ARCHIVEDATE in the tiles-specstatus.ecsv files is later than TIMESTAMP in the mtl-done-tiles.ecsv file.
Adding a --reprocess option to the run_mtl_loop script, which ensures that MTL will only operate on reprocessed tiles when prompted.
Setting up a framework to keep reprocessing completely separate from "standard" MTL looping so that the approaches can be run independently.
This facilitates running the standard MTL loop while further testing the results for reprocessed tiles.
Looser error checking that allows us to reprocess piecemeal, even if zdone flips between true and false in new ways in tiles-specstatus.ecsv.
This facilitates full QA runs for reprocessed tiles without tripping up the "standard" MTL loop.
The skeleton of the logic for updating reprocessed targets.
In the desitarget.mtl.reprocess_ledger() function.
This is the part of the code which needs more extensive testing before it will be ready for prime time.
My plan is to merge this early tomorrow morning and tag version 2.1.0 of desitarget (before my usual Tuesday teaching obligations). This should allow us to start making standard MTL updates again tomorrow.
This PR begins setting up MTL to interface with tiles that have been reprocessed by the spectroscopic pipeline. This only partially completes the required work. As it will be easiest to test the code on real-world data, the DESI Operations team signed off on merging this partial work now while I further debug the logic. This will allow us to both recommence running the "standard" MTL loop and to begin adding information for reprocessed tiles to the
tiles-specstatus.ecsv
file after they have gone through QA.This PR already achieves:
ARCHIVEDATE
in thetiles-specstatus.ecsv
files is later thanTIMESTAMP
in themtl-done-tiles.ecsv
file.--reprocess
option to therun_mtl_loop
script, which ensures that MTL will only operate on reprocessed tiles when prompted.zdone
flips betweentrue
andfalse
in new ways intiles-specstatus.ecsv
.desitarget.mtl.reprocess_ledger()
function.