coderxio / sagerx

Open drug data pipelines curated by pharmacists.
https://coderx.io/sagerx
Other
45 stars 12 forks source link

Automate transfer of updated mart flatfiles to shared file location #266

Closed jrlegrand closed 5 months ago

jrlegrand commented 6 months ago

Depends on #262

Problem Statement

Once #262 is complete, we will need to automate saving these marts off as flatfiles and transferring them to a shared file location (AWS s3 bucket, i.e.)

Criteria for Success

Marts run automatically every week and are transferred to a shared file location automatically every week.

Possibly should do a check that the mart recently ran to completion with no errors before transferring to shared file location.

Additional Information

We are still evaluating shared file location options, but are leaning towards AWS s3.

jrlegrand commented 5 months ago

Closed as part of PR #275