dbprocessing has functionality to pass in extra days for processing. For example, to create day N of a level 2 file might require context data from the previous or next day, so in addition to the day N of the level 1, dbprocessing can be configured to pass in level 1 from day N-1 and/or day N+1. (This can be extended to arbitrary number of days.)
In designing this feature, there was a certain amount of ambiguity in the spec, leading to some possible inconsistencies in when a file is processed. There are basically three decision points for dbprocessing, all in the context of having this functionality enabled for a process:
One we have decided to process day N, do we pass in day N-1/day N+1 on the command line? This is the core of the feature, and it does work, so the answer here is yes.
If we have a new day N-1/N+1 input, do we reprocess day N even if we have no new day N inputs? This currently isn't happening consistently. I think this should be yes, because otherwise there are issues of data inconsistency.
Do we make day N for the first time if we ONLY have inputs for day N-1 or day N+1, but not day N? I believe this is never happening with the current implementation. My inclination is to say no, because this is meant to support context data. For data where the timestamp is in the "wrong" day (e.g. a daily file for 2019-02-02 that really starts at 2019-02-01 23:58), the best approach would be to have the inspector properly populate the utc_start_time and utc_stop_time.
(Close issue when: definition complete and implementation updated + tested)
dbprocessing has functionality to pass in extra days for processing. For example, to create day N of a level 2 file might require context data from the previous or next day, so in addition to the day N of the level 1, dbprocessing can be configured to pass in level 1 from day N-1 and/or day N+1. (This can be extended to arbitrary number of days.)
In designing this feature, there was a certain amount of ambiguity in the spec, leading to some possible inconsistencies in when a file is processed. There are basically three decision points for dbprocessing, all in the context of having this functionality enabled for a process:
(Close issue when: definition complete and implementation updated + tested)