hubverse-org / hubverse-cloud

Test hub for S3 data submission and storage
MIT License
0 stars 0 forks source link

Automatically copy incoming S3 model-output data to our chosen organization strategy #12

Closed bsweger closed 8 months ago

bsweger commented 9 months ago

The current GitHub action we're using simply copies model-output data from it's GitHub file structure into S3. Once we decide the best organization/partitioning strategy to use for cloud-based model outputs (#11), add a step after the initial S3 sync to copy the data into the structure that will be accessed by our query tooling (e.g. hubUtils)

bsweger commented 9 months ago

This is related to #20

Before we implement any kind of "data conversion" logic, we need to agree on how it should be triggered.

bsweger commented 8 months ago

Closing this--we don't yet have a concrete idea for how we want to organize the user-facing model-output data, so no need to keep this around in the backlog. We'll open another issue when there is more information.