This PR extends #774 by including the actual, final logic for reprocessing tiles. This logic is to:
Assemble all previous observations that touch the set of reprocessed tiles including any "new" observations from the reprocessed tiles.
Find the most recent observations for unique combinations of TILEID + TARGETID and store these in a redshift catalog (which I'll refer to as the "all-zcat" below).
Determine the order in which tiles were originally processed and loop through the tiles in this original order:
Start with the UNOBS state for a given TARGETID.
Update the UNOBS state (progressively, in tile order) using entries from the "all-zcat" for a given TARGETID.
Hence, recover the final state for each TARGETID with the reprocessed observations substituted for the "orginal" observations.
The full progression of states for each TARGETID is then appended to the ledgers, includingBAD observations (which, previously, never appeared in the ledgers).
This PR also includes:
During reprocessing, TIMESTAMPs are mocked up in advance for speed.
i.e. instead of having to delay the code by 1 second for each tile so as to ensure a unique TIMESTAMP.
The numobsfromzcat flag is deprecated as a user-specified option when executing run_mtl_loop.
as we have always, in actuality, retrieved NUMOBS from the ledgers themselves rather than the redshift catalog.
it seemed increasingly fragile to retain an option we've never used.
This PR extends #774 by including the actual, final logic for reprocessing tiles. This logic is to:
TILEID
+TARGETID
and store these in a redshift catalog (which I'll refer to as the "all-zcat" below).UNOBS
state for a givenTARGETID
.UNOBS
state (progressively, in tile order) using entries from the "all-zcat" for a givenTARGETID
.TARGETID
with the reprocessed observations substituted for the "orginal" observations.TARGETID
is then appended to the ledgers, includingBAD
observations (which, previously, never appeared in the ledgers).This PR also includes:
TIMESTAMP
s are mocked up in advance for speed.TIMESTAMP
.numobsfromzcat
flag is deprecated as a user-specified option when executingrun_mtl_loop
.NUMOBS
from the ledgers themselves rather than the redshift catalog.