At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.
This epic will implement the following:
when the pipeline runs it will evaluate what data we already have and the data that is available and process based on that configuration
Separate the steps taken in the dailypipeline so that if one step fails it doesn't impact other unrelated scripts.
Reconfigure scripts so that they can automatically pickup where they last got to.
when data is downloaded immediately push it back to object storage.
before downloading data check to see if the data exists in ostore first
general improvements to try to make the code more readable. (use utility methods with descriptive names vs file name / date / data hacking using string manipulation regex's etc.) Methods should also provide clear description of what they do with examples
At the moment the script is setup to run a single date. Sometimes this doesn't work as the modis data or the viirs data is not available when the script runs.
This epic will implement the following:
Dependencies
Epic