Create GHA to injest MPOML Data

bcgov / nr-rfc-climate-obs

Transition of the existiing climate observations data pipeline to enable running off prem

Apache License 2.0

1 stars 0 forks source link

Create GHA to injest MPOML Data #11

Open franTarkenton opened 1 year ago

franTarkenton commented 1 year ago

This will be a simple pipeline that will pull the data from the federal gov datamart to a location in object store that emulates the location being used in the on prem pipeline.

The timeing for now will be the same as the schedule used on prem

franTarkenton commented 1 year ago

name of windows scheduler job: MPOML
Schedule: Every day @ 8.15am
runs: E:\Shared\MPOML\SRC\xml_daily.bat
pulls from: https://hpfx.collab.science.gc.ca//observations/xml/BC/yesterday/yesterdaybc_e.xml
output 2 = E:\Shared\MPOML\Today\<%YYYY%-%MT%-%DD%>\todaybc_e.xml
output 1 = E:\Shared\MPOML\Yesterday\<%YYYY%-%MT%-%DD%>\yesterdaybc_e.xml

franTarkenton commented 1 year ago

created the branch https://github.com/bcgov/nr-rfc-climate-obs/tree/feat/11-mpoml which now contains the code to pull the mpoml data down... just the two .xml files.
next step is to create another action to do this.

franTarkenton commented 1 year ago

Completed w/ gha.

scheduled for 8.15am every morning. currently only downloads the data.