search

uace-azmet / azmet-forecast-qa

Developing QA/QC routines for AZMet
0 stars 1 forks source link

Look into cloud storage options for targets #32

Closed Aariq closed 1 year ago

Aariq commented 1 year ago

The current github action runs the targets pipeline once a day. I think the way it works is it commits all the files generated by running the pipeline, including the .rds files in the data store, to the targets-runs branch. This is necessary so those targets are available the next time the action runs so it doesn't have to start from scratch. However, I think committing these non-diffable files every day is going to quickly balloon the size of this repo on GitHub (at least on that targets-runs branch). An alternative would be to use cloud storage for targets. @dlebauer and I think there is a way to get University-supported AWS S3 buckets, but we need to look into it.

Aariq commented 1 year ago

I realized that it is only committing the forecast targets daily, which are likely quite small. But if we want to use an S3 bucket, I should estimate the usage and contact Matt Rahr.

Aariq commented 1 year ago

Asked about S3 bucket, someone at CCT has it on their to-do list

Aariq commented 1 year ago

I no longer think this is necessary

KristinaRiemer commented 1 year ago

@Aariq why's the storage not needed anymore?

Aariq commented 1 year ago

I don't think it's actually going to use up that much space, and worst-case we can nuke the targets_runs branch along with it's git history and start fresh. I also just never heard back from the person who was going to get me a S3 bucket to try.