Closed bsweger closed 7 months ago
Got something working in the hubverse-cloud
repo: https://github.com/Infectious-Disease-Modeling-Hubs/hubverse-cloud/blob/main/.github/workflows/publish_to_s3.yaml
Will PR this to the hubverse-actions
repo.
The action is safe for adding to hubs that don't have any cloud config set up in admin.json
. If it doesn't detect the cloud config, it will skip the steps that sync data to S3.
Cloud-enabled hubs will need a github action that syncs data in the following hub folders to cloud storage (as specified in the hub's admin config):
hub-config
model-metadata
model-output
(model-output
data should sync to a "raw" cloud storage directory/folder so we can save its original state before doing any subsequent transforms)Here is an example that we've been testing with: https://github.com/Infectious-Disease-Modeling-Hubs/hubverse-cloud/blob/main/.github/workflows/publish_to_s3.yaml
role-to-assume
should be derived from a combination of the Hubverse AWS account number + hub nameaws s3 sync
commands should be derived from the hub's admin config