Reprocessing of data is outlined here.
To support this, we need several changes:
motusServer
[ ] convert the various rerun/reprocess scripts to a function that can cause a reprocessing
of one or more boot sessions from one or more receivers
[ ] record reprocessing events in the motus.reprocess table
[ ] record reprocessed boot sessions in the motus.reprocessBootSessions table
[ ] which batches are created or retired in the motus.reprocessBatches table
[ ] new dataServer API entry: reprocessing_info(X) for information on reprocessing event X
[ ] new fields replacedIn, reprocessID added to the motus.batches table. These record
which reprocessing event (if any) replaced, generated each batch.
[ ] return new fields replacedIn and reprocessID in the batches_for_* API entries
[ ] new dataServer API entry: reprocessing_batches(X) returns the batchIDs generated by
reprocessing event X
[ ] new boolean parameter unreplacedOnly to all batches_for_* dataServer API entries;
if specified and true, only batches not replaced by subsequent reprocessing are returned.
Otherwise, all batches are returned
[ ] pushToMotus() needs to be made aware of reprocessing.
motusClient
[ ] allow user to specify a policy on reprocessing in the Motus session environment:
Motus$acceptReprocessedData <- TRUE: automatically accept reprocessed data, replacing
existing data
Motus$acceptReprocessedData <- FALSE: never acccept reprocessed data
Motus$acceptReprocessedData <- NA: ask about each reprocessing event
[ ] specify unreplacedOnly on calls to batches_for_* as appropriate to this policy
[ ] check the replacedIn and reprocessID fields in records returned by batches_for_* API
calls and deal with them according to the policy above
[ ] new function showReprocess() lists reprocessing events noticed by tagme(), and their details
[ ] new function acceptReprocess(X) replaces any relevant existing data with those generated by reprocessing event X; the data replaced might be from the original, or any reprocessing.
[ ] new function rejectReprocess(X) removes any relevant existing data generated by reprocessing
event X, restoring the previous
[ ] new function cleanupReprocess() removes any retired data from the database, and vacuums it,
to reduce size.
Reprocessing of data is outlined here. To support this, we need several changes:
motusServer
motus.reprocesstablemotus.reprocessBootSessionstablemotus.reprocessBatchestablereprocessing_info(X)for information on reprocessing eventXreplacedIn,reprocessIDadded to themotus.batchestable. These record which reprocessing event (if any) replaced, generated each batch.replacedInandreprocessIDin thebatches_for_*API entriesreprocessing_batches(X)returns the batchIDs generated by reprocessing eventXunreplacedOnlyto allbatches_for_*dataServer API entries; if specified and true, only batches not replaced by subsequent reprocessing are returned.Otherwise, all batches are returned
pushToMotus()needs to be made aware of reprocessing.motusClient
Motus$acceptReprocessedData <- TRUE: automatically accept reprocessed data, replacing existing dataMotus$acceptReprocessedData <- FALSE: never acccept reprocessed dataMotus$acceptReprocessedData <- NA: ask about each reprocessing eventunreplacedOnlyon calls tobatches_for_*as appropriate to this policyreplacedInandreprocessIDfields in records returned bybatches_for_*API calls and deal with them according to the policy aboveshowReprocess()lists reprocessing events noticed bytagme(), and their detailsacceptReprocess(X)replaces any relevant existing data with those generated by reprocessing event X; the data replaced might be from the original, or any reprocessing.rejectReprocess(X)removes any relevant existing data generated by reprocessing event X, restoring the previouscleanupReprocess()removes any retired data from the database, and vacuums it, to reduce size.