search

meltano / hub

The single source of truth for all Meltano plugins, including all available Singer Taps and Targets: https://hub.meltano.com
https://hub.meltano.com
50 stars 68 forks source link

Fetch metrics from S3 when deploying Hub #567

Closed tayloramurphy closed 2 years ago

tayloramurphy commented 2 years ago

Switching to Netlify broke what we had on Pages

https://gitlab.com/meltano/legacy-ci/hub/-/blob/main/.gitlab-ci.yml#L38-39

aaronsteers commented 2 years ago

Sounds like we need to:

  1. [ ] Add S3 creds to the netlify config: https://app.netlify.com/sites/meltano-hub/settings/deploys#environment
  2. [ ] Find where Netflify gets its build instructions and try to inject these two lines: https://gitlab.com/meltano/legacy-ci/hub/-/blob/main/.gitlab-ci.yml#L38-39

Does that approach sound reasonable? @pandemicsyn, @tayloramurphy, @pnadolny13 , @alexmarple, @afolson

tayloramurphy commented 2 years ago

Part of the CI script was getting the aws cli installed as well, so as long as there are no limitiations on curl and zip that should be ok

aaronsteers commented 2 years ago

TIL: netlify.toml: https://docs.netlify.com/configure-builds/file-based-configuration/

aaronsteers commented 2 years ago

@alexmarple and @rwfeather - We considered above using AWS CLI commands as were previously part of the CI build.

However, I think we probably should just move this to the Gridsome page build process natively.

Also, spoke with @tayloramurphy and we agreed we'll skip the Jekyll side to prioritize the new Gridsome site.

tayloramurphy commented 2 years ago

@aaronsteers as part of this issue can the Hub be setup to deploy data so there's minimal delay on the metrics?

rwfeather commented 2 years ago

Getting the netlify build process to run the same AWS cli flow is feeling kinda tricky (currently experimenting with adding the steps to a makefile, changing netlify build command to make bundle)

@aaronsteers @alexmarple: If I understand correctly, the gridsome approach would be something like the data imports, correct? That seems like a viable approach to me, especially if we'll be fine not fixing this for the existing site.

I also just have a question that might simplify build-time concerns: Are the metrics.yml and audit.yml files sensitive at all? If they aren't, we could open up their permissions slightly in s3 and access them publicly at their URL (https://prod-meltano-bucket-01.s3.us-east-2.amazonaws.com/hub_metrics/audit.yml). That would let us skip the AWS CLI and do a simple curl or fetch() as part of the gridsome data import.