search

bcgov / nr-rfc-processing

River forecast centre data processing solution
Apache License 2.0
4 stars 2 forks source link

Modis / Download and process step #63

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

Sub ticket of 31...

Rework the download of modis data.

relates to: #44

franTarkenton commented 1 year ago

To get this working needed to re-work how the downloads are being performed.

The refactor of the granule downloads is complete, the next step that has been started is to include the object storage sync.

franTarkenton commented 1 year ago

Completed... Download process has been reworked to:

  1. check for the file locally. If the file exists locally then it also checks to make sure its in object storage. If not then it gets uploaded there.
  2. if the file doesn't exist locally it checks to see if its in object storage, and if so pulls from there.
  3. if the file does not exist locally AND does not exist in object storage THEN it will go to the National Snow and Ice Data Centre and pull from there, and then immediately after it will upload the file to object storage.