This is a "PR on a PR" to document the changes I made to the pipelinerefactor branch in PR #2187 while testing daily mode, and to allow @akremin an opportunity to review them before they are added to that branch. Changes are:
BUGFIX: log.log -> log.info typo
FEATURE: Added more logging to make it easier to see the boundaries between multiple invocations of desi_proc_night --daily ... when they are all piped to the same log.
FEATURE: changed "ok to proceed with submitting cals" criteria to require a 1s flat instead of requiring 3 cte flats.
Reasoning: Although we currently take 3 cte flats and thus --daily works for current data, the pipeline only requires/uses the 1s flat so this makes the pipeline job logic closer to what is actually required. This enables testing on older data and gives a little more robustness for the future if we update our CTE flat exposures in the future.
FEATURE: adds --still-acquiring option to be able to test --daily mode on nights other than the current night. If not specified, it defaults to previous logic based upon current night and observing hours.
Note: this moved the "true_night" and "still_acquiring" logic out of the "if daily:" block since "still_acquiring" has to be correctly defined even in non-daily mode, and the new logic also needs to know "true_night" while also being able to distinguish between "still_acquiring" being auto-derived (from None) vs. being overridden by the user as True/False.
TESTING: adds a unit test that only runs at NERSC that mimics daily mode by staging links to real raw data in a separate testing $DESI_SPECTRO_DATA, and runs desi_proc_night --daily --dry-run-level 1 ... multiple times as new data are staged. This test catches the BUGFIX and exercises the new FEATUREs added.
FEATURE: added desispec.test.util.link_rawdata to facilitate the new test. That code is also useful for non-unittest interactive testing to stage links (I originally wrote it for interactive testing of desi_daily_proc --daily and then ported the workflow into desispec.test).
This is a "PR on a PR" to document the changes I made to the pipelinerefactor branch in PR #2187 while testing daily mode, and to allow @akremin an opportunity to review them before they are added to that branch. Changes are:
log.log->log.infotypodesi_proc_night --daily ...when they are all piped to the same log.--still-acquiringoption to be able to test--dailymode on nights other than the current night. If not specified, it defaults to previous logic based upon current night and observing hours.desi_proc_night --daily --dry-run-level 1 ...multiple times as new data are staged. This test catches the BUGFIX and exercises the new FEATUREs added.desispec.test.util.link_rawdatato facilitate the new test. That code is also useful for non-unittest interactive testing to stage links (I originally wrote it for interactive testing ofdesi_daily_proc --dailyand then ported the workflow intodesispec.test).