search

mozilla / fxa-activity-metrics

A server for managing the Firefox Accounts metrics database and pipeline
1 stars 3 forks source link

Dockerize these scripts so we can run them from the fxa admin node #115

Closed rfk closed 5 years ago

rfk commented 5 years ago

Ref https://bugzilla.mozilla.org/show_bug.cgi?id=1513129#c5

philbooth commented 5 years ago

The skeleton of the Dockerfile is in master now, but it needs tweaking to add connection details and creds for Redshift. And the CircleCI config is in master too, it just just needs environment variables for Dockerhub creds.

Locally:

I've retained the same makefile for now so there are two separate run commands, but really we could squish them together. I think we've always run them together as a pair anyway.

When you run it currently, it fails looking for config.json because I haven't changed that stuff yet. Presumably we'll add environment variables to Circle and then it can pick up Redshift creds from there. But I didn't want to just go straight ahead and do that because I wasn't sure if we want to set up fresh creds for any reason, or if we can just continue using the same ones that the helper instance is using.

rfk commented 5 years ago

From mtg, let's update the python scripts to just pull creds from the environment rather than config.json in a similar way to our amplitude lambda jobs.

rfk commented 5 years ago

Also, looks like some email-service copy-pasta here:

https://github.com/mozilla/fxa-activity-metrics/blob/master/.circleci/config.yml#L38

jbuck commented 5 years ago

This is nearly working, there's one more thing feature that I'd like to support, which is adding Redshift IAM role support. For example, when running a COPY command you can specify an IAM role for Redshift to use via IAM_ROLE 'arn:aws:iam::123456789012:role/RedshiftCopyUnload'. To do this, we'll need to add support for specifying an IAM role via environment variable, then adding that command to the 3 places where we currently use CREDENTIALS. The problem I was running into is that CREDENTIALS is mutually exclusive with an IAM_ROLE statement and I don't know how to get templating into Python strings or if I should be using something different here

jbuck commented 5 years ago

Related ops PR: https://github.com/mozilla-services/cloudops-deployment/pull/2829

philbooth commented 5 years ago

We can close this now.