simplify workflow logic

[sbailey]

Simplify workflow logic

• daily ops "wake up, run what you can, exit, repeat" instead of long-running manager process for more natural scronjob integration
• reprocessing becomes same logic since "run what you can" is the same as "run everything" when reprocessing a night
• handle common cases of non-standard cals (e.g. two sets of cals on a single night)
• support calib overrides (see #2174), including linking to other nights as part of the automated pipeline instead of a manual before-the-prod action.
• remove Cori-era special cases
• exptable is authoritative source of what needs to be processed, proctable is authoritative source of what has been processed

pseudocode from @akremin @sbailey whiteboad discussion Jan 24:

exptable, override_info = read_exptable(night)
proctable = read_proctable(night)

# try to run ccdcalib job
# returns success=False if e.g. we don't have enough cals yet and override_info hasn't told us
#   to do something else; exit and wait for next invocation
success, ptable = do_ccdcalib(exptable, proctable, override_info, dryrun)
if not success:
    exit()

success, ptable = do_arcsflats(exptable, proctable, override_info, dryrun)
if not success:
    exit()

loop over science tiles in exptable:
    ...

[sbailey]

Implemented in PR #2187. Closing.