Make script idempotent - Githubissues

bcgov / nr-rfc-processing

River forecast centre data processing solution

Apache License 2.0

4 stars 2 forks source link

Make script idempotent #65

Closed franTarkenton closed 1 year ago

franTarkenton commented 1 year ago

When the script runs it should treat the object store as the source of truth. This task will:

break up the daily pipeline into smaller components.
- each satellite will run as a separate pipeline,
- each step in the satellite specific pipeline will be broken into smaller pieces and reflected in separate steps in the github action. Each of these steps should first check ostore for data and then pull locally, if its not available the go to source.
- steps:
  - build aoi's - creating the basin / watershed directories locally and extract the data from the shapefiles
  - download - download the data - convert into two way download and then push to ostore
  - process - process the data - push changes as data is processed. Ideally make this async
  - calculate stats - research spike, look into how this works and best way to persist the results. (think its getting saved to a sql db)
  - daily_kml - drop this step for now as its not being used
  - plot - Again create and cache at same time

franTarkenton commented 1 year ago

Put a lot of time into fixing the code in other tickets. At this stage thing its good enough. Investing more time into patching up the code further is of diminishing returns.

Objective of this refactor would be to understand clearly what data each step requires as inputs / outputs. Should check to see if the output already exists before embarking on re-creating it.